- K Nearest Neighbor algorithm
- Ensemble learning strategy
- Bagging: bootstrapping, random forest
- Boosting: Adaptive Boosting (AdaBoost), Gradient Boosting Machine (GBM) and eXtreme Gradient Boosting (XGBoost)
- R/Python demo
March 06, 2020
Outline
Overview of K Nearest Neighbor (KNN)
- Key ideas: data points closer to each other tend to behave similarly
- Lazy learner: no active model-building is done with training dataset
- Vs. active learner
- No training time, long prediction time. Too slow for an online system.
- Instance-based and distance-based learning
- Sensitive towards irrelevant attributes
- Sensitive towards outliers and noisy data
- Requires data preprocessing: conversion from categorical to numerical attributes, standardization, normalization
- Non-parametric method: no assumption of data distribution
- Could detect highly complex nonlinear decision boundary
Variance vs. Bias Tradeoff
- \(Error = Variance + Bias\)
- Variance: stability of model and predictions
- Bias: accuracy of the model/methodology
- Overfitting: high variance, low bias
- “No Free Lunch” theorem: trade off between variance and bias, no single algorithm will outperform other algorithms on all datasets
Hyper-Parameters of KNN
- Number of neighbors (K)
- Too large: high bias
- Too small: high variance (overfitting)
- Distance functions
- Euclidean
- Manhattan
- Minkowski
Demo Dataset: Diabetes
# install.packages("mlbench")
library(caret)
library(mlbench)
data("PimaIndiansDiabetes")
diabetes <- PimaIndiansDiabetes
str(diabetes)
## 'data.frame': 768 obs. of 9 variables: ## $ pregnant: num 6 1 8 1 0 5 3 10 2 8 ... ## $ glucose : num 148 85 183 89 137 116 78 115 197 125 ... ## $ pressure: num 72 66 64 66 40 74 50 0 70 96 ... ## $ triceps : num 35 29 0 23 35 0 32 0 45 0 ... ## $ insulin : num 0 0 0 94 168 0 88 0 543 0 ... ## $ mass : num 33.6 26.6 23.3 28.1 43.1 25.6 31 35.3 30.5 0 ... ## $ pedigree: num 0.627 0.351 0.672 0.167 2.288 ... ## $ age : num 50 31 32 21 33 30 26 29 53 54 ... ## $ diabetes: Factor w/ 2 levels "neg","pos": 2 1 2 1 2 1 2 1 2 2 ...
summary(diabetes)
## pregnant glucose pressure triceps ## Min. : 0.000 Min. : 0.0 Min. : 0.00 Min. : 0.00 ## 1st Qu.: 1.000 1st Qu.: 99.0 1st Qu.: 62.00 1st Qu.: 0.00 ## Median : 3.000 Median :117.0 Median : 72.00 Median :23.00 ## Mean : 3.845 Mean :120.9 Mean : 69.11 Mean :20.54 ## 3rd Qu.: 6.000 3rd Qu.:140.2 3rd Qu.: 80.00 3rd Qu.:32.00 ## Max. :17.000 Max. :199.0 Max. :122.00 Max. :99.00 ## insulin mass pedigree age ## Min. : 0.0 Min. : 0.00 Min. :0.0780 Min. :21.00 ## 1st Qu.: 0.0 1st Qu.:27.30 1st Qu.:0.2437 1st Qu.:24.00 ## Median : 30.5 Median :32.00 Median :0.3725 Median :29.00 ## Mean : 79.8 Mean :31.99 Mean :0.4719 Mean :33.24 ## 3rd Qu.:127.2 3rd Qu.:36.60 3rd Qu.:0.6262 3rd Qu.:41.00 ## Max. :846.0 Max. :67.10 Max. :2.4200 Max. :81.00 ## diabetes ## neg:500 ## pos:268 ## ## ## ##
Train and Test Baseline KNN Model
library(caret) set.seed(188) train_index <- createDataPartition(diabetes$diabetes, p = 0.7, list = FALSE) diabetes_train <- diabetes[train_index, ] diabetes_test <- diabetes[-train_index, ] model_knn <- train(diabetes ~ ., data = diabetes_train, method = "knn") predict_knn <- predict(model_knn, newdata = diabetes_test)
Measure Model Performance: Hold-Out Method
confusionMatrix(predict_knn, diabetes_test$diabetes)
## Confusion Matrix and Statistics ## ## Reference ## Prediction neg pos ## neg 126 36 ## pos 24 44 ## ## Accuracy : 0.7391 ## 95% CI : (0.6773, 0.7946) ## No Information Rate : 0.6522 ## P-Value [Acc > NIR] : 0.002949 ## ## Kappa : 0.4041 ## Mcnemar's Test P-Value : 0.155580 ## ## Sensitivity : 0.8400 ## Specificity : 0.5500 ## Pos Pred Value : 0.7778 ## Neg Pred Value : 0.6471 ## Prevalence : 0.6522 ## Detection Rate : 0.5478 ## Detection Prevalence : 0.7043 ## Balanced Accuracy : 0.6950 ## ## 'Positive' Class : neg ##
Measure Model Performance: Bootstrap Method
print(model_knn)
## k-Nearest Neighbors ## ## 538 samples ## 8 predictor ## 2 classes: 'neg', 'pos' ## ## No pre-processing ## Resampling: Bootstrapped (25 reps) ## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... ## Resampling results across tuning parameters: ## ## k Accuracy Kappa ## 5 0.6901734 0.3082508 ## 7 0.6989885 0.3185598 ## 9 0.7121062 0.3449934 ## ## Accuracy was used to select the optimal model using the largest value. ## The final value used for the model was k = 9.
Pre-process Data
pre_process <- preProcess(diabetes_train, method = c("scale", "center"))
pre_process
## Created from 538 samples and 9 variables ## ## Pre-processing: ## - centered (8) ## - ignored (1) ## - scaled (8)
diabetes_train1 <- predict(pre_process, newdata = diabetes_train) diabetes_test1 <- predict(pre_process, newdata = diabetes_test) summary(diabetes_train1)
## pregnant glucose pressure triceps ## Min. :-1.1089 Min. :-3.7915 Min. :-3.61879 Min. :-1.2893 ## 1st Qu.:-0.8243 1st Qu.:-0.6594 1st Qu.:-0.36502 1st Qu.:-1.2893 ## Median :-0.2550 Median :-0.1270 Median : 0.05482 Median : 0.1813 ## Mean : 0.0000 Mean : 0.0000 Mean : 0.00000 Mean : 0.0000 ## 3rd Qu.: 0.5989 3rd Qu.: 0.6169 3rd Qu.: 0.47466 3rd Qu.: 0.7328 ## Max. : 3.7299 Max. : 2.4414 Max. : 2.78379 Max. : 4.7771 ## insulin mass pedigree age ## Min. :-0.6787 Min. :-4.146198 Min. :-1.1951 Min. :-1.0365 ## 1st Qu.:-0.6787 1st Qu.:-0.605945 1st Qu.:-0.7106 1st Qu.:-0.7809 ## Median :-0.5087 Median : 0.009751 Median :-0.2590 Median :-0.3549 ## Mean : 0.0000 Mean : 0.000000 Mean : 0.0000 Mean : 0.0000 ## 3rd Qu.: 0.4158 3rd Qu.: 0.596587 3rd Qu.: 0.4550 3rd Qu.: 0.6675 ## Max. : 6.5131 Max. : 4.460722 Max. : 5.8088 Max. : 4.0755 ## diabetes ## neg:350 ## pos:188 ## ## ## ##
summary(diabetes_test1)
## pregnant glucose pressure triceps ## Min. :-1.10892 Min. :-3.79153 Min. :-3.6188 Min. :-1.2893 ## 1st Qu.:-0.82428 1st Qu.:-0.72207 1st Qu.:-0.2601 1st Qu.:-1.2893 ## Median :-0.25501 Median :-0.07999 Median : 0.1598 Median :-0.1251 ## Mean :-0.04834 Mean :-0.01667 Mean : 0.0263 Mean :-0.1032 ## 3rd Qu.: 0.31426 3rd Qu.: 0.55426 3rd Qu.: 0.5796 3rd Qu.: 0.6562 ## Max. : 2.59135 Max. : 2.37871 Max. : 2.1540 Max. : 1.7745 ## insulin mass pedigree ## Min. :-0.678722 Min. :-4.1462 Min. :-1.17414 ## 1st Qu.:-0.678722 1st Qu.:-0.7791 1st Qu.:-0.68668 ## Median :-0.300427 Median :-0.1442 Median :-0.39959 ## Mean :-0.001154 Mean :-0.1419 Mean :-0.05735 ## 3rd Qu.: 0.354151 3rd Qu.: 0.4074 3rd Qu.: 0.36673 ## Max. : 4.243363 Max. : 2.9087 Max. : 5.41402 ## age diabetes ## Min. :-1.03648 neg:150 ## 1st Qu.:-0.78089 pos: 80 ## Median :-0.31229 ## Mean : 0.02147 ## 3rd Qu.: 0.58230 ## Max. : 3.13827
names(pre_process)
## [1] "dim" "bc" "yj" ## [4] "et" "invHyperbolicSine" "mean" ## [7] "std" "ranges" "rotation" ## [10] "method" "thresh" "pcaComp" ## [13] "numComp" "ica" "wildcards" ## [16] "k" "knnSummary" "bagImp" ## [19] "median" "data" "rangeBounds"
typeof(pre_process)
## [1] "list"
str(pre_process)
## List of 21 ## $ dim : int [1:2] 538 9 ## $ bc : NULL ## $ yj : NULL ## $ et : NULL ## $ invHyperbolicSine: NULL ## $ mean : Named num [1:8] 3.9 121.1 69 21 79.8 ... ## ..- attr(*, "names")= chr [1:8] "pregnant" "glucose" "pressure" "triceps" ... ## $ std : Named num [1:8] 3.51 31.93 19.05 16.32 117.63 ... ## ..- attr(*, "names")= chr [1:8] "pregnant" "glucose" "pressure" "triceps" ... ## $ ranges : NULL ## $ rotation : NULL ## $ method :List of 3 ## ..$ scale : chr [1:8] "pregnant" "glucose" "pressure" "triceps" ... ## ..$ center: chr [1:8] "pregnant" "glucose" "pressure" "triceps" ... ## ..$ ignore: chr "diabetes" ## $ thresh : num 0.95 ## $ pcaComp : NULL ## $ numComp : NULL ## $ ica : NULL ## $ wildcards :List of 2 ## ..$ PCA: chr(0) ## ..$ ICA: chr(0) ## $ k : num 5 ## $ knnSummary :function (x, ...) ## $ bagImp : NULL ## $ median : NULL ## $ data : NULL ## $ rangeBounds : num [1:2] 0 1 ## - attr(*, "class")= chr "preProcess"
pre_process$method$scale[1]
## [1] "pregnant"
set.seed(188) diabetes1 <- predict(pre_process, diabetes) summary(diabetes1)
## pregnant glucose pressure ## Min. :-1.10892 Min. :-3.791531 Min. :-3.618793 ## 1st Qu.:-0.82428 1st Qu.:-0.690751 1st Qu.:-0.365020 ## Median :-0.25501 Median :-0.126972 Median : 0.159782 ## Mean :-0.01448 Mean :-0.004992 Mean : 0.007876 ## 3rd Qu.: 0.59890 3rd Qu.: 0.601241 3rd Qu.: 0.579623 ## Max. : 3.72990 Max. : 2.441351 Max. : 2.783792 ## triceps insulin mass ## Min. :-1.28931 Min. :-0.678722 Min. :-4.14620 ## 1st Qu.:-1.28931 1st Qu.:-0.678722 1st Qu.:-0.64443 ## Median : 0.12005 Median :-0.419441 Median :-0.04156 ## Mean :-0.03091 Mean :-0.000346 Mean :-0.04251 ## 3rd Qu.: 0.67153 3rd Qu.: 0.403032 3rd Qu.: 0.54849 ## Max. : 4.77705 Max. : 6.513133 Max. : 4.46072 ## pedigree age diabetes ## Min. :-1.19508 Min. :-1.036484 neg:500 ## 1st Qu.:-0.69939 1st Qu.:-0.780886 pos:268 ## Median :-0.31436 Median :-0.354891 ## Mean :-0.01717 Mean : 0.006429 ## 3rd Qu.: 0.44449 3rd Qu.: 0.667499 ## Max. : 5.80877 Max. : 4.075463
train_index1 <- createDataPartition(diabetes1$diabetes, p = 0.7, list = FALSE) diabetes_train1 <- diabetes1[train_index1, ] diabetes_test1 <- diabetes1[-train_index1, ]
New Model Using Standardized Dataset
model_knn1 <- train(diabetes ~ ., data = diabetes_train1, method = "knn") predict_knn1 <- predict(model_knn1, newdata = diabetes_test1) confusionMatrix(predict_knn1, diabetes_test1$diabetes, positive = "pos")
## Confusion Matrix and Statistics ## ## Reference ## Prediction neg pos ## neg 128 40 ## pos 22 40 ## ## Accuracy : 0.7304 ## 95% CI : (0.6682, 0.7866) ## No Information Rate : 0.6522 ## P-Value [Acc > NIR] : 0.006878 ## ## Kappa : 0.3729 ## Mcnemar's Test P-Value : 0.030850 ## ## Sensitivity : 0.5000 ## Specificity : 0.8533 ## Pos Pred Value : 0.6452 ## Neg Pred Value : 0.7619 ## Prevalence : 0.3478 ## Detection Rate : 0.1739 ## Detection Prevalence : 0.2696 ## Balanced Accuracy : 0.6767 ## ## 'Positive' Class : pos ##
Tune the KNN Model
model_knn2 <- train(diabetes ~ ., data = diabetes_train1, method = "knn",
tuneGrid = data.frame(k = seq(1, 25)),
trControl = trainControl(method = "repeatedcv",
number = 5, repeats = 3))
print(model_knn2)
## k-Nearest Neighbors ## ## 538 samples ## 8 predictor ## 2 classes: 'neg', 'pos' ## ## No pre-processing ## Resampling: Cross-Validated (5 fold, repeated 3 times) ## Summary of sample sizes: 430, 430, 431, 431, 430, 431, ... ## Resampling results across tuning parameters: ## ## k Accuracy Kappa ## 1 0.7137764 0.3664780 ## 2 0.7038191 0.3375583 ## 3 0.7298719 0.3901417 ## 4 0.7144283 0.3457516 ## 5 0.7360909 0.3933307 ## 6 0.7385831 0.3998428 ## 7 0.7484943 0.4184038 ## 8 0.7503519 0.4257615 ## 9 0.7510153 0.4271870 ## 10 0.7454540 0.4126475 ## 11 0.7466771 0.4145063 ## 12 0.7497981 0.4195199 ## 13 0.7503750 0.4217115 ## 14 0.7472828 0.4130209 ## 15 0.7540787 0.4291351 ## 16 0.7553133 0.4303757 ## 17 0.7602688 0.4408109 ## 18 0.7590516 0.4363427 ## 19 0.7633783 0.4444193 ## 20 0.7639841 0.4466424 ## 21 0.7608919 0.4368933 ## 22 0.7584285 0.4295055 ## 23 0.7652244 0.4461837 ## 24 0.7596746 0.4327782 ## 25 0.7615265 0.4353147 ## ## Accuracy was used to select the optimal model using the largest value. ## The final value used for the model was k = 23.
Sensitivity Analysis of KNN
plot(model_knn2)
Ensemble Learning Motivations
- Additive model: combine multiple classifiers together and take consensus
- Different mechanisms to produce multiple classifiers from the same datasets
- Bagging (Bootstrap AGGregating): create multiple samples (with replacement) from the original dataset and train multiple classifiers in parallel; reduce variance
- Boosting: an iterative algorithm that induces base (weak) learners sequentially (e.g. assigns different weights to the data points to focus on those tougher cases): reduce bias
- Random Forest: train large number of decision stumps (smaller decision trees with less nodes) and use committee majority voting to decide the final classification
Illustratration of Benefit of Ensemble Learning
- Wisdom of the mass
- Case study
- 25 weak learners with 30% error rate
- What is the error rate of the ensemble classifier
Bagging vs. Boosting
Bagging: Bootstrap Resampling Strategy
- Sampling with replacement
- 0.632 Rule
- Roughly \(1/3\) data won’t be included in the bootstrap sample, therefore used for validation purpose
- \(lim_{n\rightarrow\infty}(1-\frac{1}{n})^n = e^{-1}\approx 0.368\)
- Simulation
mean(sapply(createResample(1:100, times = 10000), function(x) length(unique(x))))
## [1] 63.3761
Algorithms: Random Forest (1)
Produce a distribution of simple ML models on subsets of the original data
Combine the distribution into one “aggregated” model
Train
- Pick \(k\) features from the dataset with \(m\) features (\(k \ll m\), e.g. \(\sqrt{m}\))
- Among \(k\) features, pick the one that maximizes information gain (or gain ratio) and use it as the node to further split the data
- Repeat step 2 to derive individual decision tree
- Repeat step 1 - 3 to generate \(n\) number of trees
Algorithms: Random Forest (2)
- Predict
- Apply each individual tree from the \(n\) random trees to the test data point and derive the outcomes and store the \(n\) predicted outcome
- Calculate the weight for each predicted outcome
- Consider the outcome with the highest weight as the final classication/prediction for the test data point
Adaptive Boosting (AdaBoosting) Illustration
Gradient Boosting Regressor Algorithm
GBM/XGB
- Gradient Boosting Machine (GBM)
- Additive model
- Boost performance of weak learners to strong learners additively and sequentially
- Use gradient descent to construct new base learners that minimize loss function
- Gradient descent is used in the function space, instead of in the parameter space (e.g. backpropagation in neural network)
- Extreme Gradient Boosting (XGB) improves upon GBM
- Faster using parallel computing
- Regularization (hyperparameters such as \(\alpha\), \(\lambda\)), pruning (\(\gamma\)), cross validation enabled, etc.
Analogy of GBM
Visualize GBM
Ensemble Model Hyper-Parameters
- tree-related parameters: maxdepth/max_depth, minsplit/min_samples_split, minbucket/min_samples_leaf, etc.
- mtry/max_features: optional integer for number of features to randomly select at each split, such as \(\sqrt{n}\) and \(n\) is the number of features, used for random forest or GBM.
- ntree/n_estimators: number of classifiers
- shrinkage/learning_rate (\(\eta\)): GBM/XGB, gradient descent algorithm
- objective function/loss: classification vs. regression
- subsample, etc.
Property of the Algorithms: Pros and Cons
- Pros
- Perform well on a wide variety of tasks
- Some ensemble methods such as random forest are robust to overfitting
- Cons
- More hyper-parameters to tune
- Interpretability issue
- Might be sensitive to outliers/noises and overfitting: boosting
- Higher computational cost
Bagging: Training
model_bag <- train(diabetes ~ ., data = diabetes_train1, method = "treebag") model_bag
## Bagged CART ## ## 538 samples ## 8 predictor ## 2 classes: 'neg', 'pos' ## ## No pre-processing ## Resampling: Bootstrapped (25 reps) ## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... ## Resampling results: ## ## Accuracy Kappa ## 0.7369069 0.4135043
Bagging: Prediction and Performance Evaluation
predict_bag <- predict(model_bag, newdata = diabetes_test1) confusionMatrix(predict_bag, diabetes_test1$diabetes)
## Confusion Matrix and Statistics ## ## Reference ## Prediction neg pos ## neg 128 30 ## pos 22 50 ## ## Accuracy : 0.7739 ## 95% CI : (0.7143, 0.8263) ## No Information Rate : 0.6522 ## P-Value [Acc > NIR] : 4.208e-05 ## ## Kappa : 0.4898 ## Mcnemar's Test P-Value : 0.3317 ## ## Sensitivity : 0.8533 ## Specificity : 0.6250 ## Pos Pred Value : 0.8101 ## Neg Pred Value : 0.6944 ## Prevalence : 0.6522 ## Detection Rate : 0.5565 ## Detection Prevalence : 0.6870 ## Balanced Accuracy : 0.7392 ## ## 'Positive' Class : neg ##
Random Forest
model_rf <- train(diabetes ~ ., data = diabetes_train1, method = "rf") model_rf
## Random Forest ## ## 538 samples ## 8 predictor ## 2 classes: 'neg', 'pos' ## ## No pre-processing ## Resampling: Bootstrapped (25 reps) ## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... ## Resampling results across tuning parameters: ## ## mtry Accuracy Kappa ## 2 0.7624434 0.4640791 ## 5 0.7529308 0.4486494 ## 8 0.7505543 0.4436233 ## ## Accuracy was used to select the optimal model using the largest value. ## The final value used for the model was mtry = 2.
Gradient Boosting Machine
model_gbm <- train(diabetes ~ ., data = diabetes_train1, method = "gbm")
## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.1909 nan 0.1000 0.0159 ## 2 1.1682 nan 0.1000 0.0116 ## 3 1.1445 nan 0.1000 0.0102 ## 4 1.1208 nan 0.1000 0.0095 ## 5 1.1006 nan 0.1000 0.0085 ## 6 1.0832 nan 0.1000 0.0069 ## 7 1.0691 nan 0.1000 0.0065 ## 8 1.0581 nan 0.1000 0.0044 ## 9 1.0433 nan 0.1000 0.0059 ## 10 1.0319 nan 0.1000 0.0049 ## 20 0.9391 nan 0.1000 0.0007 ## 40 0.8469 nan 0.1000 0.0010 ## 60 0.8044 nan 0.1000 0.0000 ## 80 0.7735 nan 0.1000 0.0002 ## 100 0.7475 nan 0.1000 -0.0010 ## 120 0.7305 nan 0.1000 -0.0017 ## 140 0.7150 nan 0.1000 -0.0007 ## 150 0.7078 nan 0.1000 -0.0008 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.1760 nan 0.1000 0.0209 ## 2 1.1369 nan 0.1000 0.0173 ## 3 1.1031 nan 0.1000 0.0150 ## 4 1.0708 nan 0.1000 0.0139 ## 5 1.0449 nan 0.1000 0.0100 ## 6 1.0197 nan 0.1000 0.0104 ## 7 1.0033 nan 0.1000 0.0057 ## 8 0.9841 nan 0.1000 0.0091 ## 9 0.9674 nan 0.1000 0.0043 ## 10 0.9544 nan 0.1000 0.0034 ## 20 0.8558 nan 0.1000 0.0001 ## 40 0.7502 nan 0.1000 -0.0001 ## 60 0.6847 nan 0.1000 0.0001 ## 80 0.6300 nan 0.1000 -0.0009 ## 100 0.5886 nan 0.1000 -0.0012 ## 120 0.5658 nan 0.1000 -0.0014 ## 140 0.5353 nan 0.1000 -0.0010 ## 150 0.5226 nan 0.1000 -0.0011 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.1692 nan 0.1000 0.0185 ## 2 1.1242 nan 0.1000 0.0208 ## 3 1.0760 nan 0.1000 0.0205 ## 4 1.0458 nan 0.1000 0.0110 ## 5 1.0191 nan 0.1000 0.0109 ## 6 0.9942 nan 0.1000 0.0104 ## 7 0.9688 nan 0.1000 0.0093 ## 8 0.9497 nan 0.1000 0.0041 ## 9 0.9314 nan 0.1000 0.0054 ## 10 0.9184 nan 0.1000 0.0038 ## 20 0.8053 nan 0.1000 0.0011 ## 40 0.6789 nan 0.1000 -0.0004 ## 60 0.5971 nan 0.1000 -0.0021 ## 80 0.5370 nan 0.1000 0.0002 ## 100 0.4821 nan 0.1000 -0.0006 ## 120 0.4440 nan 0.1000 -0.0010 ## 140 0.4109 nan 0.1000 -0.0007 ## 150 0.3975 nan 0.1000 -0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2359 nan 0.1000 0.0222 ## 2 1.1957 nan 0.1000 0.0177 ## 3 1.1647 nan 0.1000 0.0140 ## 4 1.1401 nan 0.1000 0.0113 ## 5 1.1160 nan 0.1000 0.0065 ## 6 1.0989 nan 0.1000 0.0078 ## 7 1.0755 nan 0.1000 0.0114 ## 8 1.0573 nan 0.1000 0.0063 ## 9 1.0437 nan 0.1000 0.0038 ## 10 1.0260 nan 0.1000 0.0067 ## 20 0.9087 nan 0.1000 0.0028 ## 40 0.8031 nan 0.1000 0.0008 ## 60 0.7492 nan 0.1000 -0.0006 ## 80 0.7100 nan 0.1000 -0.0003 ## 100 0.6792 nan 0.1000 -0.0012 ## 120 0.6601 nan 0.1000 -0.0008 ## 140 0.6443 nan 0.1000 -0.0008 ## 150 0.6398 nan 0.1000 -0.0012 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2215 nan 0.1000 0.0294 ## 2 1.1715 nan 0.1000 0.0262 ## 3 1.1339 nan 0.1000 0.0136 ## 4 1.1016 nan 0.1000 0.0141 ## 5 1.0667 nan 0.1000 0.0185 ## 6 1.0395 nan 0.1000 0.0107 ## 7 1.0144 nan 0.1000 0.0129 ## 8 0.9921 nan 0.1000 0.0086 ## 9 0.9728 nan 0.1000 0.0072 ## 10 0.9570 nan 0.1000 0.0051 ## 20 0.8194 nan 0.1000 0.0042 ## 40 0.7056 nan 0.1000 0.0009 ## 60 0.6433 nan 0.1000 -0.0015 ## 80 0.5938 nan 0.1000 -0.0002 ## 100 0.5527 nan 0.1000 -0.0013 ## 120 0.5203 nan 0.1000 -0.0009 ## 140 0.4957 nan 0.1000 -0.0009 ## 150 0.4850 nan 0.1000 -0.0020 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2139 nan 0.1000 0.0295 ## 2 1.1519 nan 0.1000 0.0274 ## 3 1.0935 nan 0.1000 0.0208 ## 4 1.0528 nan 0.1000 0.0174 ## 5 1.0234 nan 0.1000 0.0103 ## 6 0.9921 nan 0.1000 0.0136 ## 7 0.9651 nan 0.1000 0.0103 ## 8 0.9441 nan 0.1000 0.0070 ## 9 0.9194 nan 0.1000 0.0115 ## 10 0.8954 nan 0.1000 0.0075 ## 20 0.7526 nan 0.1000 0.0033 ## 40 0.6207 nan 0.1000 0.0011 ## 60 0.5451 nan 0.1000 0.0003 ## 80 0.4917 nan 0.1000 -0.0043 ## 100 0.4511 nan 0.1000 -0.0024 ## 120 0.4076 nan 0.1000 -0.0013 ## 140 0.3758 nan 0.1000 -0.0015 ## 150 0.3584 nan 0.1000 -0.0016 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2295 nan 0.1000 0.0194 ## 2 1.2032 nan 0.1000 0.0118 ## 3 1.1769 nan 0.1000 0.0112 ## 4 1.1425 nan 0.1000 0.0150 ## 5 1.1137 nan 0.1000 0.0139 ## 6 1.0856 nan 0.1000 0.0090 ## 7 1.0630 nan 0.1000 0.0103 ## 8 1.0417 nan 0.1000 0.0092 ## 9 1.0228 nan 0.1000 0.0091 ## 10 1.0055 nan 0.1000 0.0081 ## 20 0.8881 nan 0.1000 0.0037 ## 40 0.7803 nan 0.1000 -0.0001 ## 60 0.7268 nan 0.1000 -0.0002 ## 80 0.6890 nan 0.1000 -0.0009 ## 100 0.6590 nan 0.1000 -0.0006 ## 120 0.6364 nan 0.1000 -0.0007 ## 140 0.6189 nan 0.1000 -0.0002 ## 150 0.6082 nan 0.1000 -0.0012 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2016 nan 0.1000 0.0302 ## 2 1.1478 nan 0.1000 0.0240 ## 3 1.0980 nan 0.1000 0.0202 ## 4 1.0587 nan 0.1000 0.0144 ## 5 1.0259 nan 0.1000 0.0135 ## 6 0.9998 nan 0.1000 0.0116 ## 7 0.9799 nan 0.1000 0.0072 ## 8 0.9595 nan 0.1000 0.0071 ## 9 0.9356 nan 0.1000 0.0109 ## 10 0.9147 nan 0.1000 0.0074 ## 20 0.7753 nan 0.1000 0.0028 ## 40 0.6645 nan 0.1000 -0.0007 ## 60 0.5897 nan 0.1000 -0.0007 ## 80 0.5408 nan 0.1000 -0.0007 ## 100 0.5110 nan 0.1000 -0.0009 ## 120 0.4739 nan 0.1000 -0.0006 ## 140 0.4418 nan 0.1000 -0.0008 ## 150 0.4293 nan 0.1000 -0.0016 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.1942 nan 0.1000 0.0375 ## 2 1.1450 nan 0.1000 0.0224 ## 3 1.0959 nan 0.1000 0.0225 ## 4 1.0486 nan 0.1000 0.0223 ## 5 1.0081 nan 0.1000 0.0137 ## 6 0.9791 nan 0.1000 0.0137 ## 7 0.9501 nan 0.1000 0.0138 ## 8 0.9253 nan 0.1000 0.0103 ## 9 0.9043 nan 0.1000 0.0085 ## 10 0.8840 nan 0.1000 0.0068 ## 20 0.7387 nan 0.1000 0.0028 ## 40 0.5932 nan 0.1000 0.0003 ## 60 0.5177 nan 0.1000 -0.0003 ## 80 0.4564 nan 0.1000 -0.0007 ## 100 0.4066 nan 0.1000 -0.0004 ## 120 0.3626 nan 0.1000 -0.0018 ## 140 0.3256 nan 0.1000 -0.0014 ## 150 0.3108 nan 0.1000 -0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2422 nan 0.1000 0.0160 ## 2 1.2047 nan 0.1000 0.0132 ## 3 1.1779 nan 0.1000 0.0119 ## 4 1.1578 nan 0.1000 0.0097 ## 5 1.1364 nan 0.1000 0.0084 ## 6 1.1205 nan 0.1000 0.0057 ## 7 1.1006 nan 0.1000 0.0077 ## 8 1.0882 nan 0.1000 0.0035 ## 9 1.0703 nan 0.1000 0.0083 ## 10 1.0561 nan 0.1000 0.0052 ## 20 0.9595 nan 0.1000 0.0034 ## 40 0.8641 nan 0.1000 -0.0009 ## 60 0.8082 nan 0.1000 0.0008 ## 80 0.7689 nan 0.1000 0.0010 ## 100 0.7421 nan 0.1000 -0.0010 ## 120 0.7196 nan 0.1000 0.0001 ## 140 0.7042 nan 0.1000 -0.0017 ## 150 0.6962 nan 0.1000 -0.0006 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2303 nan 0.1000 0.0218 ## 2 1.1853 nan 0.1000 0.0183 ## 3 1.1524 nan 0.1000 0.0140 ## 4 1.1257 nan 0.1000 0.0125 ## 5 1.0938 nan 0.1000 0.0118 ## 6 1.0650 nan 0.1000 0.0115 ## 7 1.0442 nan 0.1000 0.0069 ## 8 1.0231 nan 0.1000 0.0062 ## 9 1.0061 nan 0.1000 0.0064 ## 10 0.9851 nan 0.1000 0.0079 ## 20 0.8589 nan 0.1000 0.0028 ## 40 0.7511 nan 0.1000 -0.0001 ## 60 0.6871 nan 0.1000 -0.0003 ## 80 0.6398 nan 0.1000 -0.0006 ## 100 0.5989 nan 0.1000 -0.0007 ## 120 0.5616 nan 0.1000 -0.0004 ## 140 0.5287 nan 0.1000 -0.0008 ## 150 0.5142 nan 0.1000 -0.0006 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2213 nan 0.1000 0.0269 ## 2 1.1700 nan 0.1000 0.0206 ## 3 1.1263 nan 0.1000 0.0178 ## 4 1.0879 nan 0.1000 0.0116 ## 5 1.0588 nan 0.1000 0.0070 ## 6 1.0334 nan 0.1000 0.0120 ## 7 1.0120 nan 0.1000 0.0071 ## 8 0.9911 nan 0.1000 0.0072 ## 9 0.9713 nan 0.1000 0.0069 ## 10 0.9546 nan 0.1000 0.0050 ## 20 0.8120 nan 0.1000 0.0023 ## 40 0.6884 nan 0.1000 0.0008 ## 60 0.6076 nan 0.1000 -0.0008 ## 80 0.5422 nan 0.1000 0.0006 ## 100 0.4957 nan 0.1000 -0.0031 ## 120 0.4550 nan 0.1000 -0.0011 ## 140 0.4214 nan 0.1000 -0.0014 ## 150 0.4046 nan 0.1000 -0.0011 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2647 nan 0.1000 0.0171 ## 2 1.2326 nan 0.1000 0.0175 ## 3 1.1997 nan 0.1000 0.0150 ## 4 1.1732 nan 0.1000 0.0107 ## 5 1.1488 nan 0.1000 0.0095 ## 6 1.1308 nan 0.1000 0.0084 ## 7 1.1102 nan 0.1000 0.0091 ## 8 1.0893 nan 0.1000 0.0092 ## 9 1.0735 nan 0.1000 0.0065 ## 10 1.0611 nan 0.1000 0.0056 ## 20 0.9684 nan 0.1000 0.0027 ## 40 0.8603 nan 0.1000 -0.0024 ## 60 0.8013 nan 0.1000 -0.0005 ## 80 0.7644 nan 0.1000 0.0008 ## 100 0.7382 nan 0.1000 -0.0012 ## 120 0.7181 nan 0.1000 -0.0014 ## 140 0.6957 nan 0.1000 -0.0004 ## 150 0.6856 nan 0.1000 -0.0011 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2501 nan 0.1000 0.0277 ## 2 1.2040 nan 0.1000 0.0216 ## 3 1.1573 nan 0.1000 0.0192 ## 4 1.1242 nan 0.1000 0.0138 ## 5 1.0948 nan 0.1000 0.0142 ## 6 1.0647 nan 0.1000 0.0092 ## 7 1.0436 nan 0.1000 0.0098 ## 8 1.0241 nan 0.1000 0.0067 ## 9 1.0036 nan 0.1000 0.0081 ## 10 0.9848 nan 0.1000 0.0060 ## 20 0.8607 nan 0.1000 0.0012 ## 40 0.7294 nan 0.1000 0.0002 ## 60 0.6621 nan 0.1000 0.0005 ## 80 0.6066 nan 0.1000 -0.0004 ## 100 0.5635 nan 0.1000 0.0009 ## 120 0.5279 nan 0.1000 -0.0006 ## 140 0.5046 nan 0.1000 -0.0017 ## 150 0.4874 nan 0.1000 -0.0001 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2419 nan 0.1000 0.0266 ## 2 1.1881 nan 0.1000 0.0210 ## 3 1.1426 nan 0.1000 0.0150 ## 4 1.1040 nan 0.1000 0.0149 ## 5 1.0589 nan 0.1000 0.0190 ## 6 1.0270 nan 0.1000 0.0155 ## 7 0.9945 nan 0.1000 0.0140 ## 8 0.9685 nan 0.1000 0.0073 ## 9 0.9444 nan 0.1000 0.0094 ## 10 0.9201 nan 0.1000 0.0087 ## 20 0.7661 nan 0.1000 0.0050 ## 40 0.6384 nan 0.1000 -0.0003 ## 60 0.5515 nan 0.1000 -0.0001 ## 80 0.4880 nan 0.1000 -0.0019 ## 100 0.4421 nan 0.1000 -0.0010 ## 120 0.4028 nan 0.1000 -0.0009 ## 140 0.3670 nan 0.1000 -0.0012 ## 150 0.3526 nan 0.1000 -0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2426 nan 0.1000 0.0159 ## 2 1.2156 nan 0.1000 0.0112 ## 3 1.1849 nan 0.1000 0.0111 ## 4 1.1677 nan 0.1000 0.0087 ## 5 1.1472 nan 0.1000 0.0085 ## 6 1.1272 nan 0.1000 0.0066 ## 7 1.1138 nan 0.1000 0.0042 ## 8 1.1004 nan 0.1000 0.0029 ## 9 1.0823 nan 0.1000 0.0074 ## 10 1.0701 nan 0.1000 0.0055 ## 20 0.9787 nan 0.1000 0.0016 ## 40 0.8770 nan 0.1000 0.0023 ## 60 0.8211 nan 0.1000 -0.0009 ## 80 0.7885 nan 0.1000 -0.0005 ## 100 0.7628 nan 0.1000 -0.0009 ## 120 0.7425 nan 0.1000 -0.0010 ## 140 0.7259 nan 0.1000 -0.0007 ## 150 0.7205 nan 0.1000 -0.0013 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2256 nan 0.1000 0.0177 ## 2 1.1876 nan 0.1000 0.0172 ## 3 1.1544 nan 0.1000 0.0124 ## 4 1.1188 nan 0.1000 0.0178 ## 5 1.0903 nan 0.1000 0.0137 ## 6 1.0637 nan 0.1000 0.0098 ## 7 1.0417 nan 0.1000 0.0053 ## 8 1.0168 nan 0.1000 0.0112 ## 9 0.9955 nan 0.1000 0.0094 ## 10 0.9749 nan 0.1000 0.0081 ## 20 0.8544 nan 0.1000 -0.0005 ## 40 0.7462 nan 0.1000 -0.0003 ## 60 0.6803 nan 0.1000 -0.0008 ## 80 0.6228 nan 0.1000 -0.0010 ## 100 0.5805 nan 0.1000 -0.0008 ## 120 0.5480 nan 0.1000 -0.0023 ## 140 0.5161 nan 0.1000 -0.0016 ## 150 0.4992 nan 0.1000 0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2150 nan 0.1000 0.0253 ## 2 1.1677 nan 0.1000 0.0183 ## 3 1.1303 nan 0.1000 0.0153 ## 4 1.0918 nan 0.1000 0.0135 ## 5 1.0555 nan 0.1000 0.0142 ## 6 1.0273 nan 0.1000 0.0110 ## 7 0.9964 nan 0.1000 0.0136 ## 8 0.9717 nan 0.1000 0.0072 ## 9 0.9522 nan 0.1000 0.0063 ## 10 0.9319 nan 0.1000 0.0063 ## 20 0.7966 nan 0.1000 -0.0002 ## 40 0.6607 nan 0.1000 0.0007 ## 60 0.5839 nan 0.1000 -0.0005 ## 80 0.5225 nan 0.1000 -0.0007 ## 100 0.4749 nan 0.1000 -0.0013 ## 120 0.4296 nan 0.1000 -0.0010 ## 140 0.3934 nan 0.1000 -0.0010 ## 150 0.3767 nan 0.1000 -0.0006 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2445 nan 0.1000 0.0173 ## 2 1.2177 nan 0.1000 0.0071 ## 3 1.1865 nan 0.1000 0.0161 ## 4 1.1538 nan 0.1000 0.0125 ## 5 1.1322 nan 0.1000 0.0105 ## 6 1.1074 nan 0.1000 0.0099 ## 7 1.0867 nan 0.1000 0.0084 ## 8 1.0659 nan 0.1000 0.0092 ## 9 1.0483 nan 0.1000 0.0073 ## 10 1.0335 nan 0.1000 0.0072 ## 20 0.9182 nan 0.1000 0.0029 ## 40 0.8013 nan 0.1000 0.0013 ## 60 0.7481 nan 0.1000 0.0006 ## 80 0.7088 nan 0.1000 -0.0000 ## 100 0.6774 nan 0.1000 -0.0008 ## 120 0.6544 nan 0.1000 -0.0006 ## 140 0.6363 nan 0.1000 -0.0020 ## 150 0.6275 nan 0.1000 -0.0023 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2286 nan 0.1000 0.0297 ## 2 1.1786 nan 0.1000 0.0222 ## 3 1.1339 nan 0.1000 0.0168 ## 4 1.0946 nan 0.1000 0.0200 ## 5 1.0567 nan 0.1000 0.0196 ## 6 1.0227 nan 0.1000 0.0149 ## 7 0.9922 nan 0.1000 0.0124 ## 8 0.9712 nan 0.1000 0.0082 ## 9 0.9547 nan 0.1000 0.0051 ## 10 0.9343 nan 0.1000 0.0093 ## 20 0.8142 nan 0.1000 0.0009 ## 40 0.6890 nan 0.1000 -0.0005 ## 60 0.6181 nan 0.1000 -0.0013 ## 80 0.5729 nan 0.1000 -0.0008 ## 100 0.5372 nan 0.1000 -0.0009 ## 120 0.5093 nan 0.1000 -0.0013 ## 140 0.4814 nan 0.1000 0.0001 ## 150 0.4668 nan 0.1000 0.0002 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2185 nan 0.1000 0.0305 ## 2 1.1590 nan 0.1000 0.0279 ## 3 1.1026 nan 0.1000 0.0250 ## 4 1.0653 nan 0.1000 0.0157 ## 5 1.0251 nan 0.1000 0.0168 ## 6 0.9930 nan 0.1000 0.0126 ## 7 0.9574 nan 0.1000 0.0138 ## 8 0.9349 nan 0.1000 0.0090 ## 9 0.9123 nan 0.1000 0.0079 ## 10 0.8938 nan 0.1000 0.0071 ## 20 0.7556 nan 0.1000 0.0018 ## 40 0.6285 nan 0.1000 0.0017 ## 60 0.5437 nan 0.1000 0.0008 ## 80 0.4897 nan 0.1000 -0.0027 ## 100 0.4432 nan 0.1000 -0.0015 ## 120 0.4012 nan 0.1000 -0.0013 ## 140 0.3691 nan 0.1000 -0.0006 ## 150 0.3552 nan 0.1000 -0.0014 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2612 nan 0.1000 0.0184 ## 2 1.2347 nan 0.1000 0.0121 ## 3 1.2035 nan 0.1000 0.0078 ## 4 1.1785 nan 0.1000 0.0123 ## 5 1.1626 nan 0.1000 0.0044 ## 6 1.1420 nan 0.1000 0.0084 ## 7 1.1232 nan 0.1000 0.0087 ## 8 1.1093 nan 0.1000 0.0059 ## 9 1.0915 nan 0.1000 0.0076 ## 10 1.0766 nan 0.1000 0.0073 ## 20 0.9662 nan 0.1000 0.0001 ## 40 0.8561 nan 0.1000 -0.0005 ## 60 0.7953 nan 0.1000 0.0002 ## 80 0.7552 nan 0.1000 0.0001 ## 100 0.7201 nan 0.1000 -0.0010 ## 120 0.6975 nan 0.1000 -0.0009 ## 140 0.6728 nan 0.1000 -0.0005 ## 150 0.6621 nan 0.1000 -0.0016 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2397 nan 0.1000 0.0241 ## 2 1.2075 nan 0.1000 0.0117 ## 3 1.1740 nan 0.1000 0.0163 ## 4 1.1376 nan 0.1000 0.0157 ## 5 1.1093 nan 0.1000 0.0130 ## 6 1.0743 nan 0.1000 0.0102 ## 7 1.0505 nan 0.1000 0.0074 ## 8 1.0308 nan 0.1000 0.0078 ## 9 1.0105 nan 0.1000 0.0084 ## 10 0.9916 nan 0.1000 0.0082 ## 20 0.8664 nan 0.1000 0.0027 ## 40 0.7387 nan 0.1000 0.0000 ## 60 0.6663 nan 0.1000 -0.0009 ## 80 0.6160 nan 0.1000 -0.0030 ## 100 0.5774 nan 0.1000 -0.0023 ## 120 0.5378 nan 0.1000 -0.0026 ## 140 0.5025 nan 0.1000 -0.0003 ## 150 0.4912 nan 0.1000 -0.0003 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2426 nan 0.1000 0.0220 ## 2 1.1857 nan 0.1000 0.0250 ## 3 1.1413 nan 0.1000 0.0169 ## 4 1.1045 nan 0.1000 0.0165 ## 5 1.0682 nan 0.1000 0.0137 ## 6 1.0411 nan 0.1000 0.0109 ## 7 1.0157 nan 0.1000 0.0078 ## 8 0.9878 nan 0.1000 0.0112 ## 9 0.9671 nan 0.1000 0.0064 ## 10 0.9459 nan 0.1000 0.0083 ## 20 0.8030 nan 0.1000 0.0036 ## 40 0.6457 nan 0.1000 -0.0000 ## 60 0.5648 nan 0.1000 -0.0014 ## 80 0.4995 nan 0.1000 -0.0005 ## 100 0.4495 nan 0.1000 -0.0004 ## 120 0.4053 nan 0.1000 -0.0011 ## 140 0.3665 nan 0.1000 -0.0012 ## 150 0.3471 nan 0.1000 -0.0001 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2772 nan 0.1000 0.0142 ## 2 1.2456 nan 0.1000 0.0116 ## 3 1.2244 nan 0.1000 0.0092 ## 4 1.2011 nan 0.1000 0.0111 ## 5 1.1845 nan 0.1000 0.0052 ## 6 1.1645 nan 0.1000 0.0096 ## 7 1.1459 nan 0.1000 0.0067 ## 8 1.1291 nan 0.1000 0.0062 ## 9 1.1147 nan 0.1000 0.0062 ## 10 1.1002 nan 0.1000 0.0059 ## 20 1.0054 nan 0.1000 -0.0006 ## 40 0.9102 nan 0.1000 0.0017 ## 60 0.8595 nan 0.1000 0.0004 ## 80 0.8295 nan 0.1000 0.0010 ## 100 0.8106 nan 0.1000 -0.0004 ## 120 0.7904 nan 0.1000 -0.0008 ## 140 0.7737 nan 0.1000 -0.0012 ## 150 0.7669 nan 0.1000 -0.0006 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2695 nan 0.1000 0.0204 ## 2 1.2261 nan 0.1000 0.0200 ## 3 1.1854 nan 0.1000 0.0161 ## 4 1.1534 nan 0.1000 0.0139 ## 5 1.1271 nan 0.1000 0.0088 ## 6 1.1076 nan 0.1000 0.0066 ## 7 1.0899 nan 0.1000 0.0045 ## 8 1.0657 nan 0.1000 0.0105 ## 9 1.0486 nan 0.1000 0.0082 ## 10 1.0318 nan 0.1000 0.0072 ## 20 0.9229 nan 0.1000 0.0019 ## 40 0.8237 nan 0.1000 -0.0022 ## 60 0.7566 nan 0.1000 -0.0007 ## 80 0.7071 nan 0.1000 -0.0023 ## 100 0.6692 nan 0.1000 -0.0018 ## 120 0.6311 nan 0.1000 -0.0001 ## 140 0.5977 nan 0.1000 -0.0024 ## 150 0.5794 nan 0.1000 0.0005 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2537 nan 0.1000 0.0266 ## 2 1.2056 nan 0.1000 0.0204 ## 3 1.1622 nan 0.1000 0.0215 ## 4 1.1194 nan 0.1000 0.0202 ## 5 1.0871 nan 0.1000 0.0149 ## 6 1.0556 nan 0.1000 0.0114 ## 7 1.0244 nan 0.1000 0.0134 ## 8 0.9970 nan 0.1000 0.0113 ## 9 0.9765 nan 0.1000 0.0047 ## 10 0.9576 nan 0.1000 0.0053 ## 20 0.8433 nan 0.1000 0.0018 ## 40 0.7328 nan 0.1000 -0.0010 ## 60 0.6583 nan 0.1000 -0.0017 ## 80 0.6013 nan 0.1000 0.0010 ## 100 0.5508 nan 0.1000 -0.0018 ## 120 0.5010 nan 0.1000 -0.0008 ## 140 0.4629 nan 0.1000 -0.0009 ## 150 0.4443 nan 0.1000 -0.0005 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2372 nan 0.1000 0.0127 ## 2 1.1988 nan 0.1000 0.0161 ## 3 1.1822 nan 0.1000 0.0056 ## 4 1.1597 nan 0.1000 0.0101 ## 5 1.1365 nan 0.1000 0.0090 ## 6 1.1139 nan 0.1000 0.0099 ## 7 1.0954 nan 0.1000 0.0083 ## 8 1.0814 nan 0.1000 0.0063 ## 9 1.0643 nan 0.1000 0.0072 ## 10 1.0480 nan 0.1000 0.0043 ## 20 0.9543 nan 0.1000 0.0011 ## 40 0.8556 nan 0.1000 -0.0001 ## 60 0.7939 nan 0.1000 0.0004 ## 80 0.7555 nan 0.1000 -0.0001 ## 100 0.7211 nan 0.1000 -0.0008 ## 120 0.6914 nan 0.1000 -0.0008 ## 140 0.6689 nan 0.1000 -0.0003 ## 150 0.6571 nan 0.1000 -0.0007 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2231 nan 0.1000 0.0191 ## 2 1.1876 nan 0.1000 0.0153 ## 3 1.1442 nan 0.1000 0.0152 ## 4 1.1119 nan 0.1000 0.0103 ## 5 1.0827 nan 0.1000 0.0120 ## 6 1.0570 nan 0.1000 0.0134 ## 7 1.0381 nan 0.1000 0.0050 ## 8 1.0186 nan 0.1000 0.0058 ## 9 1.0017 nan 0.1000 0.0036 ## 10 0.9765 nan 0.1000 0.0092 ## 20 0.8584 nan 0.1000 0.0028 ## 40 0.7313 nan 0.1000 0.0000 ## 60 0.6553 nan 0.1000 -0.0005 ## 80 0.5916 nan 0.1000 -0.0005 ## 100 0.5386 nan 0.1000 -0.0005 ## 120 0.4996 nan 0.1000 -0.0008 ## 140 0.4661 nan 0.1000 -0.0004 ## 150 0.4527 nan 0.1000 -0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2079 nan 0.1000 0.0242 ## 2 1.1598 nan 0.1000 0.0183 ## 3 1.1219 nan 0.1000 0.0162 ## 4 1.0920 nan 0.1000 0.0102 ## 5 1.0600 nan 0.1000 0.0120 ## 6 1.0326 nan 0.1000 0.0109 ## 7 1.0044 nan 0.1000 0.0116 ## 8 0.9723 nan 0.1000 0.0134 ## 9 0.9502 nan 0.1000 0.0089 ## 10 0.9351 nan 0.1000 0.0032 ## 20 0.8016 nan 0.1000 0.0049 ## 40 0.6512 nan 0.1000 0.0019 ## 60 0.5544 nan 0.1000 -0.0004 ## 80 0.4896 nan 0.1000 -0.0010 ## 100 0.4373 nan 0.1000 -0.0008 ## 120 0.3888 nan 0.1000 -0.0020 ## 140 0.3493 nan 0.1000 -0.0013 ## 150 0.3305 nan 0.1000 -0.0011 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2285 nan 0.1000 0.0190 ## 2 1.1964 nan 0.1000 0.0161 ## 3 1.1673 nan 0.1000 0.0136 ## 4 1.1409 nan 0.1000 0.0106 ## 5 1.1218 nan 0.1000 0.0072 ## 6 1.1024 nan 0.1000 0.0085 ## 7 1.0863 nan 0.1000 0.0071 ## 8 1.0711 nan 0.1000 0.0079 ## 9 1.0573 nan 0.1000 0.0062 ## 10 1.0449 nan 0.1000 0.0047 ## 20 0.9346 nan 0.1000 0.0028 ## 40 0.8340 nan 0.1000 0.0003 ## 60 0.7797 nan 0.1000 0.0000 ## 80 0.7431 nan 0.1000 -0.0005 ## 100 0.7134 nan 0.1000 -0.0007 ## 120 0.6940 nan 0.1000 -0.0011 ## 140 0.6739 nan 0.1000 -0.0008 ## 150 0.6637 nan 0.1000 -0.0008 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2197 nan 0.1000 0.0216 ## 2 1.1782 nan 0.1000 0.0177 ## 3 1.1399 nan 0.1000 0.0165 ## 4 1.1063 nan 0.1000 0.0160 ## 5 1.0809 nan 0.1000 0.0129 ## 6 1.0537 nan 0.1000 0.0081 ## 7 1.0289 nan 0.1000 0.0093 ## 8 1.0023 nan 0.1000 0.0099 ## 9 0.9815 nan 0.1000 0.0083 ## 10 0.9614 nan 0.1000 0.0074 ## 20 0.8455 nan 0.1000 0.0010 ## 40 0.7327 nan 0.1000 -0.0009 ## 60 0.6607 nan 0.1000 -0.0025 ## 80 0.6119 nan 0.1000 -0.0011 ## 100 0.5654 nan 0.1000 -0.0012 ## 120 0.5313 nan 0.1000 -0.0005 ## 140 0.4998 nan 0.1000 -0.0004 ## 150 0.4833 nan 0.1000 -0.0005 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2099 nan 0.1000 0.0271 ## 2 1.1591 nan 0.1000 0.0219 ## 3 1.1198 nan 0.1000 0.0135 ## 4 1.0789 nan 0.1000 0.0184 ## 5 1.0447 nan 0.1000 0.0151 ## 6 1.0146 nan 0.1000 0.0124 ## 7 0.9852 nan 0.1000 0.0110 ## 8 0.9612 nan 0.1000 0.0093 ## 9 0.9362 nan 0.1000 0.0087 ## 10 0.9123 nan 0.1000 0.0106 ## 20 0.7760 nan 0.1000 0.0020 ## 40 0.6340 nan 0.1000 -0.0011 ## 60 0.5591 nan 0.1000 0.0003 ## 80 0.4948 nan 0.1000 -0.0006 ## 100 0.4450 nan 0.1000 -0.0008 ## 120 0.3969 nan 0.1000 0.0001 ## 140 0.3590 nan 0.1000 -0.0012 ## 150 0.3440 nan 0.1000 -0.0015 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2572 nan 0.1000 0.0205 ## 2 1.2236 nan 0.1000 0.0182 ## 3 1.1946 nan 0.1000 0.0126 ## 4 1.1699 nan 0.1000 0.0119 ## 5 1.1464 nan 0.1000 0.0099 ## 6 1.1245 nan 0.1000 0.0104 ## 7 1.1064 nan 0.1000 0.0058 ## 8 1.0851 nan 0.1000 0.0082 ## 9 1.0655 nan 0.1000 0.0071 ## 10 1.0487 nan 0.1000 0.0075 ## 20 0.9294 nan 0.1000 0.0029 ## 40 0.8175 nan 0.1000 0.0008 ## 60 0.7573 nan 0.1000 0.0013 ## 80 0.7139 nan 0.1000 -0.0007 ## 100 0.6797 nan 0.1000 -0.0011 ## 120 0.6592 nan 0.1000 -0.0005 ## 140 0.6379 nan 0.1000 -0.0009 ## 150 0.6293 nan 0.1000 -0.0005 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2353 nan 0.1000 0.0269 ## 2 1.1910 nan 0.1000 0.0197 ## 3 1.1515 nan 0.1000 0.0185 ## 4 1.1075 nan 0.1000 0.0206 ## 5 1.0747 nan 0.1000 0.0117 ## 6 1.0407 nan 0.1000 0.0149 ## 7 1.0163 nan 0.1000 0.0126 ## 8 0.9927 nan 0.1000 0.0101 ## 9 0.9689 nan 0.1000 0.0095 ## 10 0.9505 nan 0.1000 0.0083 ## 20 0.8169 nan 0.1000 0.0035 ## 40 0.7003 nan 0.1000 -0.0007 ## 60 0.6278 nan 0.1000 -0.0009 ## 80 0.5695 nan 0.1000 0.0001 ## 100 0.5275 nan 0.1000 -0.0001 ## 120 0.4873 nan 0.1000 -0.0006 ## 140 0.4595 nan 0.1000 -0.0010 ## 150 0.4463 nan 0.1000 -0.0008 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2277 nan 0.1000 0.0356 ## 2 1.1654 nan 0.1000 0.0264 ## 3 1.1161 nan 0.1000 0.0246 ## 4 1.0778 nan 0.1000 0.0199 ## 5 1.0375 nan 0.1000 0.0164 ## 6 1.0066 nan 0.1000 0.0122 ## 7 0.9814 nan 0.1000 0.0113 ## 8 0.9538 nan 0.1000 0.0102 ## 9 0.9318 nan 0.1000 0.0093 ## 10 0.9146 nan 0.1000 0.0036 ## 20 0.7552 nan 0.1000 0.0017 ## 40 0.6225 nan 0.1000 0.0000 ## 60 0.5399 nan 0.1000 -0.0007 ## 80 0.4706 nan 0.1000 -0.0017 ## 100 0.4213 nan 0.1000 -0.0008 ## 120 0.3741 nan 0.1000 -0.0011 ## 140 0.3373 nan 0.1000 -0.0010 ## 150 0.3219 nan 0.1000 -0.0006 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.3100 nan 0.1000 0.0091 ## 2 1.2842 nan 0.1000 0.0121 ## 3 1.2561 nan 0.1000 0.0087 ## 4 1.2326 nan 0.1000 0.0104 ## 5 1.2126 nan 0.1000 0.0082 ## 6 1.1954 nan 0.1000 0.0071 ## 7 1.1749 nan 0.1000 0.0064 ## 8 1.1598 nan 0.1000 0.0065 ## 9 1.1440 nan 0.1000 0.0065 ## 10 1.1285 nan 0.1000 0.0062 ## 20 1.0324 nan 0.1000 0.0041 ## 40 0.9291 nan 0.1000 -0.0005 ## 60 0.8757 nan 0.1000 -0.0001 ## 80 0.8416 nan 0.1000 -0.0002 ## 100 0.8113 nan 0.1000 -0.0006 ## 120 0.7882 nan 0.1000 -0.0004 ## 140 0.7666 nan 0.1000 -0.0010 ## 150 0.7578 nan 0.1000 -0.0002 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.3029 nan 0.1000 0.0094 ## 2 1.2649 nan 0.1000 0.0185 ## 3 1.2261 nan 0.1000 0.0158 ## 4 1.1972 nan 0.1000 0.0117 ## 5 1.1796 nan 0.1000 0.0056 ## 6 1.1502 nan 0.1000 0.0123 ## 7 1.1287 nan 0.1000 0.0085 ## 8 1.1106 nan 0.1000 0.0038 ## 9 1.0901 nan 0.1000 0.0086 ## 10 1.0741 nan 0.1000 0.0026 ## 20 0.9387 nan 0.1000 0.0041 ## 40 0.8249 nan 0.1000 -0.0004 ## 60 0.7541 nan 0.1000 -0.0011 ## 80 0.7034 nan 0.1000 -0.0023 ## 100 0.6565 nan 0.1000 -0.0011 ## 120 0.6216 nan 0.1000 -0.0003 ## 140 0.5917 nan 0.1000 -0.0007 ## 150 0.5766 nan 0.1000 -0.0019 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2832 nan 0.1000 0.0220 ## 2 1.2263 nan 0.1000 0.0238 ## 3 1.1878 nan 0.1000 0.0195 ## 4 1.1495 nan 0.1000 0.0151 ## 5 1.1210 nan 0.1000 0.0102 ## 6 1.0899 nan 0.1000 0.0132 ## 7 1.0625 nan 0.1000 0.0109 ## 8 1.0348 nan 0.1000 0.0122 ## 9 1.0103 nan 0.1000 0.0104 ## 10 0.9904 nan 0.1000 0.0093 ## 20 0.8548 nan 0.1000 0.0006 ## 40 0.7289 nan 0.1000 0.0008 ## 60 0.6468 nan 0.1000 -0.0007 ## 80 0.5817 nan 0.1000 -0.0002 ## 100 0.5282 nan 0.1000 -0.0016 ## 120 0.4887 nan 0.1000 0.0001 ## 140 0.4547 nan 0.1000 -0.0014 ## 150 0.4359 nan 0.1000 -0.0010 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2579 nan 0.1000 0.0214 ## 2 1.2233 nan 0.1000 0.0163 ## 3 1.1886 nan 0.1000 0.0136 ## 4 1.1527 nan 0.1000 0.0112 ## 5 1.1305 nan 0.1000 0.0113 ## 6 1.1102 nan 0.1000 0.0093 ## 7 1.0901 nan 0.1000 0.0091 ## 8 1.0737 nan 0.1000 0.0059 ## 9 1.0569 nan 0.1000 0.0071 ## 10 1.0423 nan 0.1000 0.0055 ## 20 0.9276 nan 0.1000 0.0055 ## 40 0.8200 nan 0.1000 0.0006 ## 60 0.7622 nan 0.1000 -0.0001 ## 80 0.7183 nan 0.1000 0.0002 ## 100 0.6850 nan 0.1000 -0.0007 ## 120 0.6578 nan 0.1000 -0.0006 ## 140 0.6382 nan 0.1000 -0.0011 ## 150 0.6291 nan 0.1000 -0.0012 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2493 nan 0.1000 0.0236 ## 2 1.2040 nan 0.1000 0.0211 ## 3 1.1584 nan 0.1000 0.0179 ## 4 1.1219 nan 0.1000 0.0189 ## 5 1.0897 nan 0.1000 0.0137 ## 6 1.0677 nan 0.1000 0.0087 ## 7 1.0367 nan 0.1000 0.0148 ## 8 1.0092 nan 0.1000 0.0122 ## 9 0.9895 nan 0.1000 0.0078 ## 10 0.9693 nan 0.1000 0.0091 ## 20 0.8287 nan 0.1000 0.0013 ## 40 0.7065 nan 0.1000 -0.0003 ## 60 0.6430 nan 0.1000 -0.0012 ## 80 0.5885 nan 0.1000 -0.0009 ## 100 0.5544 nan 0.1000 -0.0024 ## 120 0.5190 nan 0.1000 -0.0013 ## 140 0.4901 nan 0.1000 -0.0010 ## 150 0.4738 nan 0.1000 -0.0004 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2264 nan 0.1000 0.0315 ## 2 1.1721 nan 0.1000 0.0231 ## 3 1.1255 nan 0.1000 0.0183 ## 4 1.0867 nan 0.1000 0.0131 ## 5 1.0473 nan 0.1000 0.0185 ## 6 1.0139 nan 0.1000 0.0131 ## 7 0.9784 nan 0.1000 0.0147 ## 8 0.9516 nan 0.1000 0.0101 ## 9 0.9333 nan 0.1000 0.0066 ## 10 0.9077 nan 0.1000 0.0094 ## 20 0.7627 nan 0.1000 0.0022 ## 40 0.6261 nan 0.1000 0.0009 ## 60 0.5488 nan 0.1000 -0.0019 ## 80 0.4834 nan 0.1000 -0.0010 ## 100 0.4273 nan 0.1000 -0.0005 ## 120 0.3922 nan 0.1000 0.0001 ## 140 0.3604 nan 0.1000 -0.0010 ## 150 0.3436 nan 0.1000 -0.0015 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2634 nan 0.1000 0.0125 ## 2 1.2366 nan 0.1000 0.0083 ## 3 1.2139 nan 0.1000 0.0112 ## 4 1.1952 nan 0.1000 0.0072 ## 5 1.1780 nan 0.1000 0.0069 ## 6 1.1646 nan 0.1000 0.0040 ## 7 1.1464 nan 0.1000 0.0055 ## 8 1.1342 nan 0.1000 0.0041 ## 9 1.1184 nan 0.1000 0.0065 ## 10 1.1085 nan 0.1000 0.0031 ## 20 1.0155 nan 0.1000 0.0027 ## 40 0.9164 nan 0.1000 -0.0012 ## 60 0.8576 nan 0.1000 0.0005 ## 80 0.8189 nan 0.1000 0.0001 ## 100 0.7879 nan 0.1000 -0.0003 ## 120 0.7578 nan 0.1000 -0.0005 ## 140 0.7388 nan 0.1000 -0.0009 ## 150 0.7306 nan 0.1000 -0.0011 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2520 nan 0.1000 0.0195 ## 2 1.2139 nan 0.1000 0.0167 ## 3 1.1844 nan 0.1000 0.0127 ## 4 1.1571 nan 0.1000 0.0123 ## 5 1.1327 nan 0.1000 0.0100 ## 6 1.1074 nan 0.1000 0.0091 ## 7 1.0872 nan 0.1000 0.0083 ## 8 1.0704 nan 0.1000 0.0070 ## 9 1.0531 nan 0.1000 0.0053 ## 10 1.0344 nan 0.1000 0.0072 ## 20 0.9160 nan 0.1000 0.0044 ## 40 0.7894 nan 0.1000 0.0007 ## 60 0.7221 nan 0.1000 -0.0012 ## 80 0.6653 nan 0.1000 -0.0032 ## 100 0.6300 nan 0.1000 -0.0009 ## 120 0.5908 nan 0.1000 -0.0024 ## 140 0.5511 nan 0.1000 -0.0009 ## 150 0.5320 nan 0.1000 -0.0021 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2422 nan 0.1000 0.0224 ## 2 1.1999 nan 0.1000 0.0166 ## 3 1.1602 nan 0.1000 0.0155 ## 4 1.1258 nan 0.1000 0.0099 ## 5 1.0998 nan 0.1000 0.0085 ## 6 1.0743 nan 0.1000 0.0117 ## 7 1.0476 nan 0.1000 0.0133 ## 8 1.0242 nan 0.1000 0.0071 ## 9 1.0065 nan 0.1000 0.0050 ## 10 0.9843 nan 0.1000 0.0068 ## 20 0.8498 nan 0.1000 -0.0008 ## 40 0.7109 nan 0.1000 -0.0018 ## 60 0.6267 nan 0.1000 0.0002 ## 80 0.5567 nan 0.1000 -0.0004 ## 100 0.5001 nan 0.1000 -0.0012 ## 120 0.4479 nan 0.1000 -0.0007 ## 140 0.4086 nan 0.1000 -0.0003 ## 150 0.3880 nan 0.1000 -0.0010 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2782 nan 0.1000 0.0127 ## 2 1.2534 nan 0.1000 0.0120 ## 3 1.2285 nan 0.1000 0.0064 ## 4 1.2020 nan 0.1000 0.0105 ## 5 1.1796 nan 0.1000 0.0080 ## 6 1.1620 nan 0.1000 0.0091 ## 7 1.1449 nan 0.1000 0.0065 ## 8 1.1254 nan 0.1000 0.0084 ## 9 1.1103 nan 0.1000 0.0070 ## 10 1.0955 nan 0.1000 0.0049 ## 20 0.9903 nan 0.1000 0.0019 ## 40 0.8997 nan 0.1000 0.0000 ## 60 0.8557 nan 0.1000 -0.0017 ## 80 0.8272 nan 0.1000 -0.0013 ## 100 0.8016 nan 0.1000 -0.0003 ## 120 0.7825 nan 0.1000 -0.0001 ## 140 0.7672 nan 0.1000 -0.0008 ## 150 0.7586 nan 0.1000 -0.0002 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2642 nan 0.1000 0.0123 ## 2 1.2217 nan 0.1000 0.0203 ## 3 1.1853 nan 0.1000 0.0161 ## 4 1.1510 nan 0.1000 0.0141 ## 5 1.1234 nan 0.1000 0.0126 ## 6 1.1000 nan 0.1000 0.0117 ## 7 1.0750 nan 0.1000 0.0112 ## 8 1.0574 nan 0.1000 0.0082 ## 9 1.0398 nan 0.1000 0.0086 ## 10 1.0224 nan 0.1000 0.0072 ## 20 0.9115 nan 0.1000 0.0025 ## 40 0.8141 nan 0.1000 -0.0014 ## 60 0.7526 nan 0.1000 -0.0008 ## 80 0.7045 nan 0.1000 -0.0037 ## 100 0.6551 nan 0.1000 -0.0012 ## 120 0.6241 nan 0.1000 -0.0009 ## 140 0.5920 nan 0.1000 0.0003 ## 150 0.5752 nan 0.1000 -0.0030 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2593 nan 0.1000 0.0203 ## 2 1.2084 nan 0.1000 0.0212 ## 3 1.1652 nan 0.1000 0.0178 ## 4 1.1229 nan 0.1000 0.0175 ## 5 1.0892 nan 0.1000 0.0150 ## 6 1.0622 nan 0.1000 0.0117 ## 7 1.0352 nan 0.1000 0.0114 ## 8 1.0110 nan 0.1000 0.0087 ## 9 0.9928 nan 0.1000 0.0064 ## 10 0.9764 nan 0.1000 0.0045 ## 20 0.8430 nan 0.1000 0.0005 ## 40 0.7280 nan 0.1000 -0.0020 ## 60 0.6378 nan 0.1000 0.0002 ## 80 0.5829 nan 0.1000 -0.0006 ## 100 0.5282 nan 0.1000 -0.0007 ## 120 0.4821 nan 0.1000 -0.0010 ## 140 0.4372 nan 0.1000 -0.0011 ## 150 0.4154 nan 0.1000 -0.0023 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2789 nan 0.1000 0.0200 ## 2 1.2388 nan 0.1000 0.0157 ## 3 1.2110 nan 0.1000 0.0142 ## 4 1.1846 nan 0.1000 0.0098 ## 5 1.1648 nan 0.1000 0.0085 ## 6 1.1448 nan 0.1000 0.0090 ## 7 1.1263 nan 0.1000 0.0065 ## 8 1.1111 nan 0.1000 0.0081 ## 9 1.0975 nan 0.1000 0.0066 ## 10 1.0801 nan 0.1000 0.0061 ## 20 0.9730 nan 0.1000 0.0030 ## 40 0.8593 nan 0.1000 0.0010 ## 60 0.7969 nan 0.1000 -0.0009 ## 80 0.7634 nan 0.1000 -0.0004 ## 100 0.7339 nan 0.1000 -0.0009 ## 120 0.7092 nan 0.1000 -0.0010 ## 140 0.6900 nan 0.1000 -0.0003 ## 150 0.6821 nan 0.1000 -0.0002 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2691 nan 0.1000 0.0250 ## 2 1.2231 nan 0.1000 0.0186 ## 3 1.1816 nan 0.1000 0.0179 ## 4 1.1482 nan 0.1000 0.0137 ## 5 1.1181 nan 0.1000 0.0137 ## 6 1.0913 nan 0.1000 0.0106 ## 7 1.0677 nan 0.1000 0.0086 ## 8 1.0489 nan 0.1000 0.0067 ## 9 1.0314 nan 0.1000 0.0047 ## 10 1.0146 nan 0.1000 0.0045 ## 20 0.8684 nan 0.1000 0.0021 ## 40 0.7535 nan 0.1000 0.0012 ## 60 0.6814 nan 0.1000 -0.0010 ## 80 0.6201 nan 0.1000 0.0007 ## 100 0.5640 nan 0.1000 -0.0011 ## 120 0.5307 nan 0.1000 0.0002 ## 140 0.4928 nan 0.1000 0.0005 ## 150 0.4776 nan 0.1000 -0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2590 nan 0.1000 0.0245 ## 2 1.1928 nan 0.1000 0.0280 ## 3 1.1410 nan 0.1000 0.0188 ## 4 1.1064 nan 0.1000 0.0155 ## 5 1.0703 nan 0.1000 0.0153 ## 6 1.0461 nan 0.1000 0.0088 ## 7 1.0169 nan 0.1000 0.0106 ## 8 0.9875 nan 0.1000 0.0139 ## 9 0.9624 nan 0.1000 0.0109 ## 10 0.9430 nan 0.1000 0.0040 ## 20 0.8068 nan 0.1000 0.0046 ## 40 0.6647 nan 0.1000 0.0000 ## 60 0.5730 nan 0.1000 -0.0009 ## 80 0.5041 nan 0.1000 -0.0009 ## 100 0.4555 nan 0.1000 -0.0020 ## 120 0.4125 nan 0.1000 -0.0005 ## 140 0.3729 nan 0.1000 -0.0005 ## 150 0.3554 nan 0.1000 -0.0002 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2436 nan 0.1000 0.0201 ## 2 1.2065 nan 0.1000 0.0155 ## 3 1.1736 nan 0.1000 0.0135 ## 4 1.1476 nan 0.1000 0.0130 ## 5 1.1213 nan 0.1000 0.0110 ## 6 1.0980 nan 0.1000 0.0096 ## 7 1.0783 nan 0.1000 0.0077 ## 8 1.0592 nan 0.1000 0.0066 ## 9 1.0437 nan 0.1000 0.0065 ## 10 1.0279 nan 0.1000 0.0059 ## 20 0.9196 nan 0.1000 0.0041 ## 40 0.8246 nan 0.1000 0.0008 ## 60 0.7798 nan 0.1000 -0.0009 ## 80 0.7499 nan 0.1000 -0.0017 ## 100 0.7305 nan 0.1000 -0.0006 ## 120 0.7143 nan 0.1000 -0.0007 ## 140 0.6980 nan 0.1000 -0.0006 ## 150 0.6921 nan 0.1000 -0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2222 nan 0.1000 0.0253 ## 2 1.1776 nan 0.1000 0.0205 ## 3 1.1347 nan 0.1000 0.0212 ## 4 1.1013 nan 0.1000 0.0159 ## 5 1.0692 nan 0.1000 0.0137 ## 6 1.0337 nan 0.1000 0.0129 ## 7 1.0070 nan 0.1000 0.0119 ## 8 0.9844 nan 0.1000 0.0087 ## 9 0.9614 nan 0.1000 0.0100 ## 10 0.9420 nan 0.1000 0.0070 ## 20 0.8237 nan 0.1000 0.0011 ## 40 0.7267 nan 0.1000 0.0012 ## 60 0.6674 nan 0.1000 -0.0006 ## 80 0.6127 nan 0.1000 -0.0005 ## 100 0.5756 nan 0.1000 -0.0032 ## 120 0.5396 nan 0.1000 -0.0007 ## 140 0.5045 nan 0.1000 0.0010 ## 150 0.4905 nan 0.1000 -0.0005 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2132 nan 0.1000 0.0320 ## 2 1.1602 nan 0.1000 0.0252 ## 3 1.1178 nan 0.1000 0.0208 ## 4 1.0743 nan 0.1000 0.0192 ## 5 1.0353 nan 0.1000 0.0182 ## 6 1.0024 nan 0.1000 0.0142 ## 7 0.9826 nan 0.1000 0.0066 ## 8 0.9542 nan 0.1000 0.0113 ## 9 0.9291 nan 0.1000 0.0114 ## 10 0.9083 nan 0.1000 0.0090 ## 20 0.7794 nan 0.1000 -0.0013 ## 40 0.6690 nan 0.1000 -0.0013 ## 60 0.5880 nan 0.1000 0.0005 ## 80 0.5262 nan 0.1000 -0.0021 ## 100 0.4818 nan 0.1000 -0.0027 ## 120 0.4359 nan 0.1000 -0.0005 ## 140 0.3978 nan 0.1000 -0.0006 ## 150 0.3757 nan 0.1000 -0.0004 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2568 nan 0.1000 0.0165 ## 2 1.2267 nan 0.1000 0.0116 ## 3 1.2035 nan 0.1000 0.0103 ## 4 1.1801 nan 0.1000 0.0099 ## 5 1.1550 nan 0.1000 0.0117 ## 6 1.1347 nan 0.1000 0.0094 ## 7 1.1123 nan 0.1000 0.0101 ## 8 1.0944 nan 0.1000 0.0071 ## 9 1.0768 nan 0.1000 0.0060 ## 10 1.0599 nan 0.1000 0.0079 ## 20 0.9458 nan 0.1000 0.0024 ## 40 0.8225 nan 0.1000 0.0004 ## 60 0.7595 nan 0.1000 0.0007 ## 80 0.7241 nan 0.1000 -0.0001 ## 100 0.6884 nan 0.1000 -0.0006 ## 120 0.6598 nan 0.1000 0.0000 ## 140 0.6432 nan 0.1000 -0.0009 ## 150 0.6311 nan 0.1000 -0.0007 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2417 nan 0.1000 0.0215 ## 2 1.1974 nan 0.1000 0.0201 ## 3 1.1613 nan 0.1000 0.0180 ## 4 1.1194 nan 0.1000 0.0155 ## 5 1.0929 nan 0.1000 0.0086 ## 6 1.0655 nan 0.1000 0.0107 ## 7 1.0418 nan 0.1000 0.0085 ## 8 1.0172 nan 0.1000 0.0104 ## 9 0.9931 nan 0.1000 0.0100 ## 10 0.9769 nan 0.1000 0.0026 ## 20 0.8336 nan 0.1000 0.0033 ## 40 0.7080 nan 0.1000 0.0012 ## 60 0.6319 nan 0.1000 -0.0012 ## 80 0.5870 nan 0.1000 -0.0004 ## 100 0.5433 nan 0.1000 -0.0009 ## 120 0.5094 nan 0.1000 -0.0006 ## 140 0.4816 nan 0.1000 -0.0017 ## 150 0.4636 nan 0.1000 -0.0014 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2329 nan 0.1000 0.0279 ## 2 1.1825 nan 0.1000 0.0229 ## 3 1.1362 nan 0.1000 0.0214 ## 4 1.0914 nan 0.1000 0.0184 ## 5 1.0456 nan 0.1000 0.0198 ## 6 1.0104 nan 0.1000 0.0165 ## 7 0.9780 nan 0.1000 0.0125 ## 8 0.9578 nan 0.1000 0.0066 ## 9 0.9310 nan 0.1000 0.0114 ## 10 0.9107 nan 0.1000 0.0093 ## 20 0.7713 nan 0.1000 0.0035 ## 40 0.6200 nan 0.1000 0.0010 ## 60 0.5375 nan 0.1000 -0.0005 ## 80 0.4820 nan 0.1000 -0.0010 ## 100 0.4328 nan 0.1000 -0.0004 ## 120 0.3833 nan 0.1000 -0.0008 ## 140 0.3506 nan 0.1000 -0.0009 ## 150 0.3385 nan 0.1000 -0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2609 nan 0.1000 0.0169 ## 2 1.2312 nan 0.1000 0.0129 ## 3 1.2029 nan 0.1000 0.0120 ## 4 1.1815 nan 0.1000 0.0097 ## 5 1.1573 nan 0.1000 0.0080 ## 6 1.1349 nan 0.1000 0.0108 ## 7 1.1190 nan 0.1000 0.0055 ## 8 1.1104 nan 0.1000 0.0023 ## 9 1.0985 nan 0.1000 0.0035 ## 10 1.0868 nan 0.1000 0.0034 ## 20 0.9885 nan 0.1000 0.0037 ## 40 0.8822 nan 0.1000 -0.0001 ## 60 0.8275 nan 0.1000 -0.0006 ## 80 0.7855 nan 0.1000 0.0011 ## 100 0.7504 nan 0.1000 -0.0003 ## 120 0.7280 nan 0.1000 -0.0002 ## 140 0.7046 nan 0.1000 -0.0002 ## 150 0.6945 nan 0.1000 0.0006 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2519 nan 0.1000 0.0199 ## 2 1.2139 nan 0.1000 0.0180 ## 3 1.1725 nan 0.1000 0.0176 ## 4 1.1418 nan 0.1000 0.0120 ## 5 1.1130 nan 0.1000 0.0116 ## 6 1.0870 nan 0.1000 0.0129 ## 7 1.0667 nan 0.1000 0.0094 ## 8 1.0470 nan 0.1000 0.0062 ## 9 1.0283 nan 0.1000 0.0067 ## 10 1.0153 nan 0.1000 0.0026 ## 20 0.9005 nan 0.1000 0.0016 ## 40 0.7712 nan 0.1000 0.0023 ## 60 0.6961 nan 0.1000 -0.0006 ## 80 0.6427 nan 0.1000 0.0002 ## 100 0.5955 nan 0.1000 -0.0005 ## 120 0.5556 nan 0.1000 -0.0002 ## 140 0.5267 nan 0.1000 -0.0013 ## 150 0.5102 nan 0.1000 0.0000 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2389 nan 0.1000 0.0257 ## 2 1.2016 nan 0.1000 0.0162 ## 3 1.1631 nan 0.1000 0.0162 ## 4 1.1238 nan 0.1000 0.0177 ## 5 1.0851 nan 0.1000 0.0141 ## 6 1.0518 nan 0.1000 0.0152 ## 7 1.0273 nan 0.1000 0.0080 ## 8 1.0077 nan 0.1000 0.0070 ## 9 0.9879 nan 0.1000 0.0075 ## 10 0.9667 nan 0.1000 0.0094 ## 20 0.8262 nan 0.1000 0.0037 ## 40 0.6861 nan 0.1000 0.0012 ## 60 0.6028 nan 0.1000 -0.0012 ## 80 0.5415 nan 0.1000 0.0001 ## 100 0.4907 nan 0.1000 -0.0012 ## 120 0.4470 nan 0.1000 -0.0006 ## 140 0.4088 nan 0.1000 -0.0013 ## 150 0.3939 nan 0.1000 -0.0020 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2563 nan 0.1000 0.0166 ## 2 1.2269 nan 0.1000 0.0128 ## 3 1.2003 nan 0.1000 0.0115 ## 4 1.1788 nan 0.1000 0.0089 ## 5 1.1604 nan 0.1000 0.0074 ## 6 1.1380 nan 0.1000 0.0102 ## 7 1.1231 nan 0.1000 0.0069 ## 8 1.1038 nan 0.1000 0.0081 ## 9 1.0887 nan 0.1000 0.0057 ## 10 1.0736 nan 0.1000 0.0059 ## 20 0.9776 nan 0.1000 0.0009 ## 40 0.8959 nan 0.1000 0.0004 ## 60 0.8459 nan 0.1000 0.0006 ## 80 0.8084 nan 0.1000 -0.0016 ## 100 0.7821 nan 0.1000 -0.0004 ## 120 0.7659 nan 0.1000 -0.0016 ## 140 0.7463 nan 0.1000 -0.0030 ## 150 0.7349 nan 0.1000 -0.0002 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2383 nan 0.1000 0.0225 ## 2 1.2018 nan 0.1000 0.0137 ## 3 1.1596 nan 0.1000 0.0193 ## 4 1.1261 nan 0.1000 0.0133 ## 5 1.1002 nan 0.1000 0.0118 ## 6 1.0785 nan 0.1000 0.0080 ## 7 1.0506 nan 0.1000 0.0098 ## 8 1.0344 nan 0.1000 0.0045 ## 9 1.0171 nan 0.1000 0.0067 ## 10 1.0020 nan 0.1000 0.0056 ## 20 0.8889 nan 0.1000 0.0008 ## 40 0.7821 nan 0.1000 -0.0007 ## 60 0.7061 nan 0.1000 -0.0008 ## 80 0.6534 nan 0.1000 -0.0017 ## 100 0.6159 nan 0.1000 -0.0002 ## 120 0.5770 nan 0.1000 -0.0001 ## 140 0.5446 nan 0.1000 -0.0004 ## 150 0.5337 nan 0.1000 -0.0004 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2360 nan 0.1000 0.0206 ## 2 1.1917 nan 0.1000 0.0169 ## 3 1.1432 nan 0.1000 0.0183 ## 4 1.1081 nan 0.1000 0.0091 ## 5 1.0792 nan 0.1000 0.0119 ## 6 1.0566 nan 0.1000 0.0076 ## 7 1.0364 nan 0.1000 0.0055 ## 8 1.0114 nan 0.1000 0.0086 ## 9 0.9937 nan 0.1000 0.0045 ## 10 0.9720 nan 0.1000 0.0093 ## 20 0.8454 nan 0.1000 0.0010 ## 40 0.7206 nan 0.1000 0.0015 ## 60 0.6329 nan 0.1000 -0.0021 ## 80 0.5678 nan 0.1000 0.0006 ## 100 0.5141 nan 0.1000 -0.0020 ## 120 0.4736 nan 0.1000 -0.0011 ## 140 0.4390 nan 0.1000 -0.0008 ## 150 0.4218 nan 0.1000 -0.0014 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2648 nan 0.1000 0.0180 ## 2 1.2312 nan 0.1000 0.0137 ## 3 1.2068 nan 0.1000 0.0103 ## 4 1.1821 nan 0.1000 0.0117 ## 5 1.1600 nan 0.1000 0.0099 ## 6 1.1389 nan 0.1000 0.0083 ## 7 1.1196 nan 0.1000 0.0068 ## 8 1.1002 nan 0.1000 0.0067 ## 9 1.0819 nan 0.1000 0.0084 ## 10 1.0663 nan 0.1000 0.0048 ## 20 0.9508 nan 0.1000 0.0035 ## 40 0.8332 nan 0.1000 0.0008 ## 60 0.7786 nan 0.1000 -0.0007 ## 80 0.7404 nan 0.1000 0.0003 ## 100 0.7162 nan 0.1000 -0.0001 ## 120 0.6991 nan 0.1000 -0.0005 ## 140 0.6807 nan 0.1000 -0.0004 ## 150 0.6713 nan 0.1000 -0.0013 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2467 nan 0.1000 0.0222 ## 2 1.1930 nan 0.1000 0.0216 ## 3 1.1597 nan 0.1000 0.0156 ## 4 1.1297 nan 0.1000 0.0123 ## 5 1.1045 nan 0.1000 0.0080 ## 6 1.0734 nan 0.1000 0.0130 ## 7 1.0457 nan 0.1000 0.0131 ## 8 1.0195 nan 0.1000 0.0099 ## 9 0.9926 nan 0.1000 0.0103 ## 10 0.9715 nan 0.1000 0.0091 ## 20 0.8481 nan 0.1000 0.0025 ## 40 0.7336 nan 0.1000 -0.0009 ## 60 0.6640 nan 0.1000 0.0003 ## 80 0.6135 nan 0.1000 -0.0008 ## 100 0.5759 nan 0.1000 -0.0011 ## 120 0.5450 nan 0.1000 -0.0010 ## 140 0.5113 nan 0.1000 -0.0016 ## 150 0.4968 nan 0.1000 -0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2324 nan 0.1000 0.0277 ## 2 1.1793 nan 0.1000 0.0256 ## 3 1.1341 nan 0.1000 0.0182 ## 4 1.0928 nan 0.1000 0.0187 ## 5 1.0558 nan 0.1000 0.0150 ## 6 1.0260 nan 0.1000 0.0114 ## 7 0.9984 nan 0.1000 0.0086 ## 8 0.9725 nan 0.1000 0.0115 ## 9 0.9481 nan 0.1000 0.0074 ## 10 0.9310 nan 0.1000 0.0066 ## 20 0.7871 nan 0.1000 0.0029 ## 40 0.6495 nan 0.1000 -0.0012 ## 60 0.5733 nan 0.1000 -0.0012 ## 80 0.5160 nan 0.1000 -0.0010 ## 100 0.4648 nan 0.1000 0.0004 ## 120 0.4265 nan 0.1000 -0.0008 ## 140 0.3894 nan 0.1000 -0.0007 ## 150 0.3758 nan 0.1000 -0.0016 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2025 nan 0.1000 0.0215 ## 2 1.1719 nan 0.1000 0.0154 ## 3 1.1426 nan 0.1000 0.0131 ## 4 1.1151 nan 0.1000 0.0099 ## 5 1.0900 nan 0.1000 0.0091 ## 6 1.0635 nan 0.1000 0.0124 ## 7 1.0400 nan 0.1000 0.0091 ## 8 1.0235 nan 0.1000 0.0062 ## 9 1.0086 nan 0.1000 0.0052 ## 10 0.9924 nan 0.1000 0.0073 ## 20 0.8834 nan 0.1000 0.0041 ## 40 0.7690 nan 0.1000 -0.0001 ## 60 0.7087 nan 0.1000 -0.0004 ## 80 0.6679 nan 0.1000 -0.0024 ## 100 0.6371 nan 0.1000 -0.0001 ## 120 0.6145 nan 0.1000 -0.0007 ## 140 0.5955 nan 0.1000 -0.0005 ## 150 0.5871 nan 0.1000 -0.0006 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.1767 nan 0.1000 0.0301 ## 2 1.1290 nan 0.1000 0.0231 ## 3 1.0853 nan 0.1000 0.0181 ## 4 1.0528 nan 0.1000 0.0131 ## 5 1.0167 nan 0.1000 0.0163 ## 6 0.9869 nan 0.1000 0.0147 ## 7 0.9596 nan 0.1000 0.0115 ## 8 0.9401 nan 0.1000 0.0098 ## 9 0.9229 nan 0.1000 0.0053 ## 10 0.8995 nan 0.1000 0.0088 ## 20 0.7653 nan 0.1000 0.0023 ## 40 0.6421 nan 0.1000 -0.0013 ## 60 0.5662 nan 0.1000 0.0011 ## 80 0.5132 nan 0.1000 -0.0010 ## 100 0.4712 nan 0.1000 0.0001 ## 120 0.4390 nan 0.1000 -0.0008 ## 140 0.4068 nan 0.1000 0.0000 ## 150 0.3905 nan 0.1000 -0.0013 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.1769 nan 0.1000 0.0318 ## 2 1.1159 nan 0.1000 0.0280 ## 3 1.0755 nan 0.1000 0.0168 ## 4 1.0260 nan 0.1000 0.0187 ## 5 0.9906 nan 0.1000 0.0145 ## 6 0.9568 nan 0.1000 0.0117 ## 7 0.9253 nan 0.1000 0.0126 ## 8 0.9017 nan 0.1000 0.0079 ## 9 0.8775 nan 0.1000 0.0077 ## 10 0.8617 nan 0.1000 0.0049 ## 20 0.7064 nan 0.1000 0.0005 ## 40 0.5585 nan 0.1000 -0.0010 ## 60 0.4777 nan 0.1000 -0.0019 ## 80 0.4134 nan 0.1000 -0.0003 ## 100 0.3644 nan 0.1000 -0.0003 ## 120 0.3244 nan 0.1000 -0.0018 ## 140 0.2873 nan 0.1000 -0.0007 ## 150 0.2734 nan 0.1000 -0.0009 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2284 nan 0.1000 0.0183 ## 2 1.1931 nan 0.1000 0.0148 ## 3 1.1676 nan 0.1000 0.0129 ## 4 1.1426 nan 0.1000 0.0107 ## 5 1.1228 nan 0.1000 0.0084 ## 6 1.1029 nan 0.1000 0.0082 ## 7 1.0870 nan 0.1000 0.0075 ## 8 1.0685 nan 0.1000 0.0080 ## 9 1.0492 nan 0.1000 0.0073 ## 10 1.0378 nan 0.1000 0.0056 ## 20 0.9290 nan 0.1000 0.0022 ## 40 0.8378 nan 0.1000 -0.0012 ## 60 0.7934 nan 0.1000 -0.0005 ## 80 0.7569 nan 0.1000 -0.0001 ## 100 0.7296 nan 0.1000 -0.0003 ## 120 0.7053 nan 0.1000 -0.0010 ## 140 0.6888 nan 0.1000 -0.0020 ## 150 0.6816 nan 0.1000 0.0000 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2134 nan 0.1000 0.0266 ## 2 1.1647 nan 0.1000 0.0193 ## 3 1.1292 nan 0.1000 0.0166 ## 4 1.1002 nan 0.1000 0.0144 ## 5 1.0717 nan 0.1000 0.0132 ## 6 1.0460 nan 0.1000 0.0111 ## 7 1.0206 nan 0.1000 0.0086 ## 8 0.9947 nan 0.1000 0.0078 ## 9 0.9768 nan 0.1000 0.0073 ## 10 0.9594 nan 0.1000 0.0082 ## 20 0.8513 nan 0.1000 0.0007 ## 40 0.7508 nan 0.1000 -0.0008 ## 60 0.6899 nan 0.1000 -0.0009 ## 80 0.6505 nan 0.1000 -0.0028 ## 100 0.6084 nan 0.1000 -0.0009 ## 120 0.5730 nan 0.1000 0.0003 ## 140 0.5397 nan 0.1000 -0.0012 ## 150 0.5235 nan 0.1000 -0.0010 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2042 nan 0.1000 0.0309 ## 2 1.1517 nan 0.1000 0.0223 ## 3 1.1126 nan 0.1000 0.0185 ## 4 1.0742 nan 0.1000 0.0110 ## 5 1.0387 nan 0.1000 0.0169 ## 6 1.0125 nan 0.1000 0.0100 ## 7 0.9804 nan 0.1000 0.0128 ## 8 0.9528 nan 0.1000 0.0104 ## 9 0.9367 nan 0.1000 0.0030 ## 10 0.9196 nan 0.1000 0.0067 ## 20 0.8011 nan 0.1000 0.0005 ## 40 0.6806 nan 0.1000 -0.0030 ## 60 0.6035 nan 0.1000 -0.0007 ## 80 0.5338 nan 0.1000 0.0001 ## 100 0.4840 nan 0.1000 0.0001 ## 120 0.4423 nan 0.1000 -0.0001 ## 140 0.4054 nan 0.1000 -0.0024 ## 150 0.3857 nan 0.1000 -0.0003 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2318 nan 0.1000 0.0202 ## 2 1.2005 nan 0.1000 0.0169 ## 3 1.1628 nan 0.1000 0.0157 ## 4 1.1381 nan 0.1000 0.0108 ## 5 1.1145 nan 0.1000 0.0115 ## 6 1.0978 nan 0.1000 0.0066 ## 7 1.0756 nan 0.1000 0.0087 ## 8 1.0600 nan 0.1000 0.0047 ## 9 1.0400 nan 0.1000 0.0078 ## 10 1.0250 nan 0.1000 0.0059 ## 20 0.9103 nan 0.1000 0.0031 ## 40 0.8089 nan 0.1000 0.0005 ## 60 0.7597 nan 0.1000 0.0003 ## 80 0.7312 nan 0.1000 -0.0015 ## 100 0.7104 nan 0.1000 -0.0010 ## 120 0.6934 nan 0.1000 -0.0006 ## 140 0.6802 nan 0.1000 -0.0013 ## 150 0.6738 nan 0.1000 -0.0007 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2229 nan 0.1000 0.0229 ## 2 1.1765 nan 0.1000 0.0215 ## 3 1.1334 nan 0.1000 0.0206 ## 4 1.0974 nan 0.1000 0.0129 ## 5 1.0633 nan 0.1000 0.0143 ## 6 1.0397 nan 0.1000 0.0093 ## 7 1.0134 nan 0.1000 0.0083 ## 8 0.9956 nan 0.1000 0.0057 ## 9 0.9749 nan 0.1000 0.0054 ## 10 0.9524 nan 0.1000 0.0069 ## 20 0.8266 nan 0.1000 0.0019 ## 40 0.7255 nan 0.1000 -0.0004 ## 60 0.6666 nan 0.1000 0.0003 ## 80 0.6120 nan 0.1000 -0.0001 ## 100 0.5734 nan 0.1000 0.0000 ## 120 0.5368 nan 0.1000 -0.0008 ## 140 0.5100 nan 0.1000 -0.0012 ## 150 0.4936 nan 0.1000 -0.0014 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2121 nan 0.1000 0.0271 ## 2 1.1583 nan 0.1000 0.0214 ## 3 1.1084 nan 0.1000 0.0232 ## 4 1.0725 nan 0.1000 0.0151 ## 5 1.0381 nan 0.1000 0.0134 ## 6 1.0046 nan 0.1000 0.0161 ## 7 0.9742 nan 0.1000 0.0136 ## 8 0.9481 nan 0.1000 0.0090 ## 9 0.9263 nan 0.1000 0.0076 ## 10 0.9084 nan 0.1000 0.0043 ## 20 0.7616 nan 0.1000 0.0015 ## 40 0.6442 nan 0.1000 -0.0013 ## 60 0.5641 nan 0.1000 -0.0003 ## 80 0.4952 nan 0.1000 -0.0010 ## 100 0.4487 nan 0.1000 -0.0012 ## 120 0.4094 nan 0.1000 -0.0019 ## 140 0.3703 nan 0.1000 -0.0010 ## 150 0.3519 nan 0.1000 -0.0001 ## ## Iter TrainDeviance ValidDeviance StepSize Improve ## 1 1.2539 nan 0.1000 0.0143 ## 2 1.2203 nan 0.1000 0.0123 ## 3 1.1915 nan 0.1000 0.0086 ## 4 1.1714 nan 0.1000 0.0071 ## 5 1.1518 nan 0.1000 0.0082 ## 6 1.1367 nan 0.1000 0.0050 ## 7 1.1164 nan 0.1000 0.0078 ## 8 1.0996 nan 0.1000 0.0070 ## 9 1.0862 nan 0.1000 0.0049 ## 10 1.0735 nan 0.1000 0.0053 ## 20 0.9764 nan 0.1000 0.0005 ## 40 0.8873 nan 0.1000 0.0007 ## 50 0.8661 nan 0.1000 -0.0020
model_gbm
## Stochastic Gradient Boosting ## ## 538 samples ## 8 predictor ## 2 classes: 'neg', 'pos' ## ## No pre-processing ## Resampling: Bootstrapped (25 reps) ## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... ## Resampling results across tuning parameters: ## ## interaction.depth n.trees Accuracy Kappa ## 1 50 0.7581561 0.4428866 ## 1 100 0.7575227 0.4488587 ## 1 150 0.7531729 0.4404393 ## 2 50 0.7561162 0.4433477 ## 2 100 0.7484656 0.4298475 ## 2 150 0.7432515 0.4181766 ## 3 50 0.7562096 0.4490012 ## 3 100 0.7474033 0.4302205 ## 3 150 0.7441665 0.4238704 ## ## Tuning parameter 'shrinkage' was held constant at a value of 0.1 ## ## Tuning parameter 'n.minobsinnode' was held constant at a value of 10 ## Accuracy was used to select the optimal model using the largest value. ## The final values used for the model were n.trees = 50, interaction.depth ## = 1, shrinkage = 0.1 and n.minobsinnode = 10.
Gradient Boosting Machine
model_xgb <- train(diabetes ~ ., data = diabetes_train1, method = "xgbTree") model_xgb
## eXtreme Gradient Boosting ## ## 538 samples ## 8 predictor ## 2 classes: 'neg', 'pos' ## ## No pre-processing ## Resampling: Bootstrapped (25 reps) ## Summary of sample sizes: 538, 538, 538, 538, 538, 538, ... ## Resampling results across tuning parameters: ## ## eta max_depth colsample_bytree subsample nrounds Accuracy ## 0.3 1 0.6 0.50 50 0.7543059 ## 0.3 1 0.6 0.50 100 0.7514498 ## 0.3 1 0.6 0.50 150 0.7401059 ## 0.3 1 0.6 0.75 50 0.7629041 ## 0.3 1 0.6 0.75 100 0.7596995 ## 0.3 1 0.6 0.75 150 0.7528011 ## 0.3 1 0.6 1.00 50 0.7609537 ## 0.3 1 0.6 1.00 100 0.7582121 ## 0.3 1 0.6 1.00 150 0.7499080 ## 0.3 1 0.8 0.50 50 0.7595069 ## 0.3 1 0.8 0.50 100 0.7468614 ## 0.3 1 0.8 0.50 150 0.7458352 ## 0.3 1 0.8 0.75 50 0.7612828 ## 0.3 1 0.8 0.75 100 0.7504202 ## 0.3 1 0.8 0.75 150 0.7451137 ## 0.3 1 0.8 1.00 50 0.7615568 ## 0.3 1 0.8 1.00 100 0.7575015 ## 0.3 1 0.8 1.00 150 0.7529885 ## 0.3 2 0.6 0.50 50 0.7445060 ## 0.3 2 0.6 0.50 100 0.7324920 ## 0.3 2 0.6 0.50 150 0.7248227 ## 0.3 2 0.6 0.75 50 0.7502667 ## 0.3 2 0.6 0.75 100 0.7414999 ## 0.3 2 0.6 0.75 150 0.7308230 ## 0.3 2 0.6 1.00 50 0.7497999 ## 0.3 2 0.6 1.00 100 0.7341151 ## 0.3 2 0.6 1.00 150 0.7307433 ## 0.3 2 0.8 0.50 50 0.7463895 ## 0.3 2 0.8 0.50 100 0.7310613 ## 0.3 2 0.8 0.50 150 0.7255107 ## 0.3 2 0.8 0.75 50 0.7445795 ## 0.3 2 0.8 0.75 100 0.7349672 ## 0.3 2 0.8 0.75 150 0.7294988 ## 0.3 2 0.8 1.00 50 0.7505633 ## 0.3 2 0.8 1.00 100 0.7409459 ## 0.3 2 0.8 1.00 150 0.7330636 ## 0.3 3 0.6 0.50 50 0.7449172 ## 0.3 3 0.6 0.50 100 0.7327773 ## 0.3 3 0.6 0.50 150 0.7290412 ## 0.3 3 0.6 0.75 50 0.7341227 ## 0.3 3 0.6 0.75 100 0.7208463 ## 0.3 3 0.6 0.75 150 0.7195923 ## 0.3 3 0.6 1.00 50 0.7390111 ## 0.3 3 0.6 1.00 100 0.7326315 ## 0.3 3 0.6 1.00 150 0.7283834 ## 0.3 3 0.8 0.50 50 0.7339048 ## 0.3 3 0.8 0.50 100 0.7253577 ## 0.3 3 0.8 0.50 150 0.7249229 ## 0.3 3 0.8 0.75 50 0.7382761 ## 0.3 3 0.8 0.75 100 0.7275341 ## 0.3 3 0.8 0.75 150 0.7253066 ## 0.3 3 0.8 1.00 50 0.7441233 ## 0.3 3 0.8 1.00 100 0.7384999 ## 0.3 3 0.8 1.00 150 0.7325560 ## 0.4 1 0.6 0.50 50 0.7594764 ## 0.4 1 0.6 0.50 100 0.7499864 ## 0.4 1 0.6 0.50 150 0.7393398 ## 0.4 1 0.6 0.75 50 0.7583405 ## 0.4 1 0.6 0.75 100 0.7484413 ## 0.4 1 0.6 0.75 150 0.7431933 ## 0.4 1 0.6 1.00 50 0.7638681 ## 0.4 1 0.6 1.00 100 0.7542095 ## 0.4 1 0.6 1.00 150 0.7467963 ## 0.4 1 0.8 0.50 50 0.7560829 ## 0.4 1 0.8 0.50 100 0.7433300 ## 0.4 1 0.8 0.50 150 0.7384965 ## 0.4 1 0.8 0.75 50 0.7589632 ## 0.4 1 0.8 0.75 100 0.7478994 ## 0.4 1 0.8 0.75 150 0.7425844 ## 0.4 1 0.8 1.00 50 0.7602074 ## 0.4 1 0.8 1.00 100 0.7526321 ## 0.4 1 0.8 1.00 150 0.7477816 ## 0.4 2 0.6 0.50 50 0.7407292 ## 0.4 2 0.6 0.50 100 0.7296555 ## 0.4 2 0.6 0.50 150 0.7231783 ## 0.4 2 0.6 0.75 50 0.7434259 ## 0.4 2 0.6 0.75 100 0.7339772 ## 0.4 2 0.6 0.75 150 0.7284584 ## 0.4 2 0.6 1.00 50 0.7442404 ## 0.4 2 0.6 1.00 100 0.7355150 ## 0.4 2 0.6 1.00 150 0.7301150 ## 0.4 2 0.8 0.50 50 0.7334168 ## 0.4 2 0.8 0.50 100 0.7255845 ## 0.4 2 0.8 0.50 150 0.7147372 ## 0.4 2 0.8 0.75 50 0.7389953 ## 0.4 2 0.8 0.75 100 0.7303682 ## 0.4 2 0.8 0.75 150 0.7268248 ## 0.4 2 0.8 1.00 50 0.7442663 ## 0.4 2 0.8 1.00 100 0.7342825 ## 0.4 2 0.8 1.00 150 0.7287683 ## 0.4 3 0.6 0.50 50 0.7359378 ## 0.4 3 0.6 0.50 100 0.7255934 ## 0.4 3 0.6 0.50 150 0.7280381 ## 0.4 3 0.6 0.75 50 0.7385790 ## 0.4 3 0.6 0.75 100 0.7274135 ## 0.4 3 0.6 0.75 150 0.7209702 ## 0.4 3 0.6 1.00 50 0.7383441 ## 0.4 3 0.6 1.00 100 0.7288983 ## 0.4 3 0.6 1.00 150 0.7260075 ## 0.4 3 0.8 0.50 50 0.7316068 ## 0.4 3 0.8 0.50 100 0.7225762 ## 0.4 3 0.8 0.50 150 0.7234279 ## 0.4 3 0.8 0.75 50 0.7327961 ## 0.4 3 0.8 0.75 100 0.7265637 ## 0.4 3 0.8 0.75 150 0.7189373 ## 0.4 3 0.8 1.00 50 0.7353974 ## 0.4 3 0.8 1.00 100 0.7250317 ## 0.4 3 0.8 1.00 150 0.7230962 ## Kappa ## 0.4462596 ## 0.4429630 ## 0.4200132 ## 0.4658151 ## 0.4604554 ## 0.4484303 ## 0.4592086 ## 0.4581448 ## 0.4411794 ## 0.4564908 ## 0.4333714 ## 0.4326394 ## 0.4617991 ## 0.4408186 ## 0.4317274 ## 0.4598021 ## 0.4565921 ## 0.4482462 ## 0.4303652 ## 0.4084279 ## 0.3906327 ## 0.4429770 ## 0.4263209 ## 0.4031319 ## 0.4407674 ## 0.4085965 ## 0.4036242 ## 0.4374084 ## 0.4046743 ## 0.3930692 ## 0.4319423 ## 0.4125149 ## 0.4007124 ## 0.4442622 ## 0.4265787 ## 0.4107550 ## 0.4324777 ## 0.4065217 ## 0.3980458 ## 0.4093924 ## 0.3827307 ## 0.3791252 ## 0.4174869 ## 0.4064532 ## 0.3981495 ## 0.4083027 ## 0.3920819 ## 0.3903599 ## 0.4169325 ## 0.3953971 ## 0.3905689 ## 0.4316228 ## 0.4194146 ## 0.4072687 ## 0.4598474 ## 0.4423625 ## 0.4186239 ## 0.4571191 ## 0.4381275 ## 0.4291049 ## 0.4676431 ## 0.4505231 ## 0.4348747 ## 0.4497447 ## 0.4299946 ## 0.4199297 ## 0.4587197 ## 0.4376588 ## 0.4291174 ## 0.4594020 ## 0.4467976 ## 0.4373434 ## 0.4236511 ## 0.4031415 ## 0.3887923 ## 0.4296547 ## 0.4117075 ## 0.3993140 ## 0.4293416 ## 0.4146539 ## 0.4047353 ## 0.4085791 ## 0.3918990 ## 0.3670191 ## 0.4200514 ## 0.4046337 ## 0.3974451 ## 0.4300328 ## 0.4111720 ## 0.4001600 ## 0.4127821 ## 0.3919866 ## 0.3963679 ## 0.4192566 ## 0.3950423 ## 0.3808712 ## 0.4182499 ## 0.3987796 ## 0.3927267 ## 0.4073487 ## 0.3872865 ## 0.3889411 ## 0.4100441 ## 0.3944626 ## 0.3777648 ## 0.4144144 ## 0.3909891 ## 0.3878530 ## ## Tuning parameter 'gamma' was held constant at a value of 0 ## ## Tuning parameter 'min_child_weight' was held constant at a value of 1 ## Accuracy was used to select the optimal model using the largest value. ## The final values used for the model were nrounds = 50, max_depth = 1, ## eta = 0.4, gamma = 0, colsample_bytree = 0.6, min_child_weight = 1 ## and subsample = 1.
Check the Importance of Attributes
varimp_rf <- varImp(model_rf) varimp_rf
## rf variable importance ## ## Overall ## glucose 100.0000 ## mass 51.8062 ## age 40.3086 ## pedigree 28.6711 ## pregnant 11.7756 ## pressure 9.9492 ## triceps 0.4362 ## insulin 0.0000
Visualize the Attributes’ Importance
plot(varimp_rf, main = "Variable Importance with Random Forest")
Compare the Performance of Multiple Algorithms
model_comparison <- resamples(list(RF = model_rf, BAG = model_bag, KNN = model_knn,
gbm = model_gbm, xgb = model_xgb))
summary(model_comparison)
## ## Call: ## summary.resamples(object = model_comparison) ## ## Models: RF, BAG, KNN, gbm, xgb ## Number of resamples: 25 ## ## Accuracy ## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's ## RF 0.7247191 0.7487179 0.7653061 0.7624434 0.7777778 0.8088235 0 ## BAG 0.6945813 0.7258065 0.7411168 0.7369069 0.7512195 0.7817259 0 ## KNN 0.6153846 0.6887755 0.7168950 0.7121062 0.7433155 0.7817259 0 ## gbm 0.6787565 0.7389163 0.7606383 0.7581561 0.7794872 0.8031088 0 ## xgb 0.6881188 0.7512438 0.7692308 0.7638681 0.7817259 0.8115942 0 ## ## Kappa ## Min. 1st Qu. Median Mean 3rd Qu. Max. NA's ## RF 0.3724277 0.4234770 0.4718310 0.4640791 0.5022061 0.5508130 0 ## BAG 0.3077769 0.3760277 0.4247151 0.4135043 0.4444444 0.5044170 0 ## KNN 0.1355872 0.2960546 0.3634009 0.3449934 0.4008293 0.4835935 0 ## gbm 0.3096804 0.4007092 0.4512535 0.4428866 0.4813896 0.5407064 0 ## xgb 0.2940197 0.4287147 0.4817761 0.4676431 0.5098131 0.5910956 0
Graphically Compare Performance
scales <- list(x = list(relation = "free"),
y = list(relation = "free"))
bwplot(model_comparison, scales = scales)
Load in Necessary Python packages
import numpy as np import pandas as pd from sklearn.model_selection import train_test_split, ShuffleSplit, cross_val_score from sklearn.neighbors import KNeighborsClassifier from sklearn import metrics from scipy.spatial.distance import cdist from sklearn.pipeline import make_pipeline import matplotlib.pyplot as plt
Data Preparation and KNN modeling
diabetes = pd.read_csv("https://datahub.io/machine-learning/diabetes/r/diabetes.csv")
X = diabetes.drop('class', axis=1)
y = diabetes['class']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=16)
classifier = KNeighborsClassifier()
classifier.fit(X_train, y_train)
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=5, p=2, ## weights='uniform')
y_pred = classifier.predict(X_test)
print(f"Accuracy: {round(metrics.accuracy_score(y_test, y_pred)*100, 2)}%")
## Accuracy: 72.29%
df_confusion = pd.crosstab(y_test, y_pred) df_confusion
## col_0 tested_negative tested_positive ## class ## tested_negative 124 23 ## tested_positive 41 43
Preprocess Data: Standardization/Normalization
import warnings
warnings.filterwarnings("ignore")
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
## StandardScaler(copy=True, with_mean=True, with_std=True)
X_train_std = scaler.transform(X_train) X_test_std = scaler.transform(X_test) pd.DataFrame(X_train_std).mean()
## 0 -2.439596e-17 ## 1 1.868979e-16 ## 2 7.153392e-17 ## 3 5.933594e-17 ## 4 1.281822e-17 ## 5 -2.456136e-16 ## 6 8.042398e-17 ## 7 4.507051e-17 ## dtype: float64
X_train_std.std()
## 1.0
X_train.mean()
## preg 3.780261 ## plas 120.109870 ## pres 68.823091 ## skin 21.016760 ## insu 81.446927 ## mass 32.067784 ## pedi 0.475715 ## age 32.957169 ## dtype: float64
X_train.std()
## preg 3.309324 ## plas 32.972357 ## pres 19.563244 ## skin 16.123755 ## insu 115.457871 ## mass 8.165811 ## pedi 0.332525 ## age 11.459534 ## dtype: float64
KNN Performance Evaluation
classifier_std = KNeighborsClassifier() classifier_std.fit(X_train_std, y_train)
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=5, p=2, ## weights='uniform')
y_pred_std = classifier_std.predict(X_test_std) pd.Series(y_pred_std).value_counts()
## tested_negative 165 ## tested_positive 66 ## dtype: int64
value, count = np.unique(y_pred_std, return_counts=True)
pd.DataFrame({"value": value, "count": count})
## value count ## 0 tested_negative 165 ## 1 tested_positive 66
print(f"Accuracy: {round(metrics.accuracy_score(y_test, y_pred_std)*100, 2)}%")
## Accuracy: 74.89%
Use Pipeline to Streamline the Analysis
knn_pipe = make_pipeline(StandardScaler(), KNeighborsClassifier()) knn_pipe.fit(X_train, y_train)
## Pipeline(memory=None,
## steps=[('standardscaler',
## StandardScaler(copy=True, with_mean=True, with_std=True)),
## ('kneighborsclassifier',
## KNeighborsClassifier(algorithm='auto', leaf_size=30,
## metric='minkowski', metric_params=None,
## n_jobs=None, n_neighbors=5, p=2,
## weights='uniform'))],
## verbose=False)
pipe_pred = knn_pipe.predict(X_test) pd.Series(pipe_pred).value_counts()
## tested_negative 165 ## tested_positive 66 ## dtype: int64
Get Repeated Hold Out Accurary of Model
cv = ShuffleSplit(n_splits=100, test_size=0.3, random_state=16) from sklearn.model_selection import KFold cv = KFold(n_splits=10, shuffle=True, random_state=16) cross_val_score(knn_pipe, X_train, y_train, cv=cv).mean()
## 0.7022361984626136
Sensitivity Analysis
from matplotlib.legend_handler import HandlerLine2D neighbors = list(range(1, 30)) train_results = [] test_results = [] for n in neighbors: model = KNeighborsClassifier(n_neighbors=n) model.fit(X_train_std, y_train) train_pred = model.predict(X_train_std) acc = cross_val_score(model, X_train_std, y_train, cv=cv).mean()*100 train_results.append(acc) y_pred = model.predict(X_test_std) acc_test = round(metrics.accuracy_score(y_test, y_pred)*100, 2) test_results.append(acc_test)
## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=1, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=2, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=3, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=4, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=5, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=6, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=7, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=8, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=9, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=10, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=11, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=12, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=13, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=14, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=15, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=16, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=17, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=18, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=19, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=20, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=21, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=22, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=23, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=24, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=25, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=26, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=27, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=28, p=2, ## weights='uniform') ## KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski', ## metric_params=None, n_jobs=None, n_neighbors=29, p=2, ## weights='uniform')
line1, = plt.plot(neighbors, train_results, 'b', label="Train Accuracy")
line2, = plt.plot(neighbors, test_results, 'r', label="Test Accuracy")
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("Accuracy")
plt.xlabel("n_neighbors")
plt.show()
Ensemble Learning - Bagging
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
cart = DecisionTreeClassifier()
num_trees = 100
model = BaggingClassifier(base_estimator=cart, n_estimators=num_trees, random_state=16)
results = cross_val_score(model, X_train_std, y_train, cv=cv)
print(f"Accuracy: {round(results.mean()*100, 2)}%")
## Accuracy: 71.73%
Random Forest
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=num_trees, max_features=5, random_state=16)
results = cross_val_score(model, X_train_std, y_train, cv=cv)
print(f"Accuracy: {round(results.mean()*100, 2)}%")
## Accuracy: 73.59%
model
## RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini', ## max_depth=None, max_features=5, max_leaf_nodes=None, ## min_impurity_decrease=0.0, min_impurity_split=None, ## min_samples_leaf=1, min_samples_split=2, ## min_weight_fraction_leaf=0.0, n_estimators=100, ## n_jobs=None, oob_score=False, random_state=16, verbose=0, ## warm_start=False)
Feature Importance
model.fit(X_train_std, y_train)
## RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini', ## max_depth=None, max_features=5, max_leaf_nodes=None, ## min_impurity_decrease=0.0, min_impurity_split=None, ## min_samples_leaf=1, min_samples_split=2, ## min_weight_fraction_leaf=0.0, n_estimators=100, ## n_jobs=None, oob_score=False, random_state=16, verbose=0, ## warm_start=False)
feature_imp = pd.DataFrame(model.feature_importances_, index=X_train.columns,
columns=['importance']).sort_values('importance', ascending=False)
feature_imp
## importance ## plas 0.299555 ## mass 0.192756 ## pedi 0.133711 ## age 0.105525 ## pres 0.083997 ## preg 0.067676 ## insu 0.059420 ## skin 0.057359
AdaBoosting
from sklearn.ensemble import AdaBoostClassifier
model = AdaBoostClassifier(n_estimators=num_trees, random_state=16)
results = cross_val_score(model, X_train_std, y_train, cv=cv)
print(f"Accuracy: {round(results.mean()*100, 2)}%")
## Accuracy: 72.08%
model
## AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1.0, ## n_estimators=100, random_state=16)
GBM
from sklearn.ensemble import GradientBoostingClassifier as gbm
model = gbm(n_estimators=num_trees, random_state=16)
results = cross_val_score(model, X_train_std, y_train, cv=cv)
print(f"Accuracy for GBM: {round(results.mean()*100, 2)}%")
## Accuracy for GBM: 72.47%
model
## GradientBoostingClassifier(criterion='friedman_mse', init=None, ## learning_rate=0.1, loss='deviance', max_depth=3, ## max_features=None, max_leaf_nodes=None, ## min_impurity_decrease=0.0, min_impurity_split=None, ## min_samples_leaf=1, min_samples_split=2, ## min_weight_fraction_leaf=0.0, n_estimators=100, ## n_iter_no_change=None, presort='auto', ## random_state=16, subsample=1.0, tol=0.0001, ## validation_fraction=0.1, verbose=0, ## warm_start=False)
Model Tuning
from sklearn.model_selection import GridSearchCV
param_grid = {'learning_rate': np.arange(0.02, 0.1, 0.02),
'n_estimators': range(60, 160, 50),
'max_depth': range(2, 5)}
clf = GridSearchCV(gbm(), param_grid)
clf.fit(X_train_std, y_train)
## GridSearchCV(cv='warn', error_score='raise-deprecating',
## estimator=GradientBoostingClassifier(criterion='friedman_mse',
## init=None, learning_rate=0.1,
## loss='deviance', max_depth=3,
## max_features=None,
## max_leaf_nodes=None,
## min_impurity_decrease=0.0,
## min_impurity_split=None,
## min_samples_leaf=1,
## min_samples_split=2,
## min_weight_fraction_leaf=0.0,
## n_estimators=100,
## n_iter_no_change=None,
## presort='auto',
## random_state=None,
## subsample=1.0, tol=0.0001,
## validation_fraction=0.1,
## verbose=0, warm_start=False),
## iid='warn', n_jobs=None,
## param_grid={'learning_rate': array([0.02, 0.04, 0.06, 0.08]),
## 'max_depth': range(2, 5),
## 'n_estimators': range(60, 160, 50)},
## pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
## scoring=None, verbose=0)
Model Tuning (2)
print(f"Accuracy for best GBM: {round(clf.best_score_*100, 2)}%")
## Accuracy for best GBM: 75.23%
for key, val in clf.best_params_.items():
print(f"Best hyperparameter is {key}: {val}")
## Best hyperparameter is learning_rate: 0.02 ## Best hyperparameter is max_depth: 2 ## Best hyperparameter is n_estimators: 110